Method and system using biological information detection device using second light from target onto which dots formed by first light are projected to detect status of living body

ABSTRACT

A system including a light source that, in operation, projects dots onto a target, the dots being formed by first light; a first photodetector that, in operation, detects second light resulting from the projection of the dots onto the target; and a circuit that in operation, performs an individual authentication, wherein the individual authentication includes at least the following step(i) and step(ii): step(i) determining whether the target is a living body or not based on the second light, and step(ii) performing a biometric authentication of the target.

This application is a Continuation application of U.S. patentapplication Ser. No. 17/180,636, filed on Feb. 19, 2021, which is aContinuation application of U.S. patent application Ser. No. 16/845,708,filed on Apr. 10, 2020, now U.S. Pat. No. 10,959,628, which is aDivisional application of U.S. patent application Ser. No. 15/628,679,filed on Jun. 21, 2017, now U.S. Pat. No. 10,653,328, which claims thebenefit of Japanese Application No. 2016-130137, filed on Jun. 30, 2016,the disclosures of which are incorporated herein by reference.

BACKGROUND 1. Technical Field

The present disclosure relates to biological information detectiondevices. The present disclosure relates to, for example, a biologicalinformation detection device that detects biological information, suchas heartbeat, in a non-contact manner.

2. Description of the Related Art

Heartbeat, blood flow, blood pressure, blood oxygen saturation, etc. arewidely used as basic parameters for determining the health condition ofa person. These pieces of biological information relating to blood aretypically measured by using contact-type measuring instruments. Sincecontact-type measuring instruments are attached to the body of asubject, measurement, especially, long continuous measurement, sometimesincurs the subject's discomfort.

Various attempts have been made to easily obtain basic biologicalinformation for determining the health condition of a person throughmeasurement. For example, Japanese Unexamined Patent ApplicationPublication No. 2005-218507 discloses a method for detecting heart ratein a non-contact manner on the basis of image information of a face orthe like obtained with a camera. Japanese Unexamined Patent ApplicationPublication (Translation of PCT application) No. 2003-517342 discloses amethod for measuring, using a white light source and a laser lightsource, blood oxygen saturation on the basis of a laser Doppler effectof laser light scattered behind the surface of a living body. JapaneseUnexamined Patent Application Publication (Translation of PCTapplication) No. 2014-527863 discloses a method for measuring, using anordinary color camera, blood oxygen saturation while removing theinfluence of ambient light.

SUMMARY

In one general aspect, the techniques disclosed here feature a systemincluding a light source that, in operation, projects dots onto atarget, the dots being formed by first light; a first photodetectorthat, in operation, detects second light resulting from the projectionof the dots onto the target; and a circuit that in operation, performsan individual authentication, wherein the individual authenticationincludes at least the following step(i) and step(ii): step(i)determining whether the target is a living body or not based on thesecond light, and step(ii) performing a biometric authentication of thetarget.

It should be noted that general or specific embodiments may beimplemented as an element, a device, a system, a method, an integratedcircuit, a computer program, a recording medium, or any selectivecombination thereof.

Additional benefits and advantages of the disclosed embodiments willbecome apparent from the specification and drawings. The benefits and/oradvantages may be individually obtained by the various embodiments andfeatures of the specification and drawings, which need not all beprovided in order to obtain one or more of such benefits and/oradvantages.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a diagram for explaining a basic concept of how biologicalinformation is obtained in accordance with an embodiment of the presentdisclosure;

FIG. 1B is a diagram for explaining characteristics of an image of thesurface of a living body, obtained with an image capturing device;

FIG. 2 is a diagram illustrating a configuration of a biologicalinformation detection device according to a first embodiment;

FIG. 3A is a diagram illustrating an example of a configuration of theimage capturing device and examples of an image and biologicalinformation that are output in accordance with the first embodiment;

FIG. 3B is a block diagram illustrating a configuration of a computer inaccordance with the first embodiment;

FIG. 3C is a diagram for explaining a respiration sensing method inaccordance with the first embodiment;

FIG. 4A is a first diagram illustrating an example of a human bodydetection experiment in accordance with the first embodiment;

FIG. 4B is a second diagram illustrating the example of the human bodydetection experiment in accordance with the first embodiment;

FIG. 5 is a diagram illustrating a contrast calculation method used inhuman body detection in accordance with the first embodiment;

FIG. 6 is a flowchart illustrating a flow of an image processing processin accordance with the first embodiment;

FIG. 7A is a diagram schematically illustrating a biological informationdetection device and a processing process of the biological informationdetection device in accordance with a third embodiment;

FIG. 7B is a diagram for explaining an algorithm of a watch system inaccordance with the third embodiment;

FIG. 7C is a flowchart illustrating the algorithm of the watch system inaccordance with the third embodiment;

FIG. 8 is a diagram illustrating a configuration of a biologicalinformation detection device according to a fourth embodiment;

FIG. 9 is a diagram illustrating the overview of biological informationsensing that uses two image capturing devices in accordance with thefourth embodiment;

FIG. 10 is a diagram illustrating transmittance characteristics of twobandpass filters in accordance with the fourth embodiment;

FIG. 11 is a diagram illustrating examples of pulse waves measured byusing a method according to the fourth embodiment;

FIG. 12 is a diagram illustrating results obtained by measuring bloodoxygen saturation by using the method according to the fourth embodimentand a method of the related art;

FIG. 13 is a diagram illustrating a configuration of astereo-camera-type biological information detection device according tothe fourth embodiment;

FIG. 14 is a diagram illustrating a configuration of a stereo-lens-typebiological information detection device according to a fifth embodiment;

FIG. 15A is a diagram illustrating a result obtained by performingstress sensing by using a biological information detection deviceaccording to a sixth embodiment;

FIG. 15B is a diagram illustrating a nose portion and a cheek portion inan image in accordance with the sixth embodiment;

FIG. 15C is a diagram illustrating a change in blood flow and a changein blood oxygen saturation that are obtained by using the biologicalinformation detection device according to the sixth embodiment;

FIG. 16 is a cross-sectional view schematically illustrating aconfiguration of a biological information detection device according toa seventh embodiment;

FIG. 17A is a diagram illustrating a nose portion and a forehead portionin an image in accordance with the seventh embodiment;

FIG. 17B is a diagram illustrating a temporal change in total blood flow(oxyhemoglobin and deoxyhemoglobin) and a temporal change inoxyhemoglobin blood flow (oxygen saturation) when an emotion of laughteris induced in accordance with the seventh embodiment;

FIG. 18 is a diagram illustrating a relationship among emotions, totalblood flow, and blood oxygen saturation;

FIG. 19A is a diagram schematically illustrating a configuration of abiological information detection device according to an eighthembodiment;

FIG. 19B is a diagram illustrating a plurality of color filters inaccordance with the eighth embodiment;

FIG. 19C is a diagram illustrating an example of images generated inaccordance with the eighth embodiment;

FIG. 20A is a diagram illustrating a configuration of a biologicalinformation detection device according to a ninth embodiment;

FIG. 20B is a diagram illustrating a plurality of color filters inaccordance with the ninth embodiment;

FIG. 20C is a diagram illustrating an example of images generated inaccordance with the ninth embodiment;

FIG. 20D is a diagram illustrating an example of a configuration of amulti-spectral sensor including four types of color filters for red (R),green (G), blue (B), and infrared (IR) light;

FIG. 21 is a diagram illustrating an example (comparative example) of aconfiguration of a biological information sensing system using an imagecapturing device; and

FIG. 22 is a diagram illustrating absorption coefficients and scatteringcoefficients of hemoglobin, melanin, and water, which are maincomponents of a living body, in a wavelength range from visible light tonear-infrared light

DETAILED DESCRIPTION

Prior to description of embodiments of the present disclosure,underlying knowledge forming the basis of the present disclosure will bedescribed.

Since remote biological information sensing with a camera enables longcontinuous measurement without causing the subject to feel constrained,various applications of such remote biological information sensing areanticipated. For example, at medical facilities such as hospitals,applications such as constantly monitoring the condition of a patientand quickly dealing with a sudden change in condition and utilizing dataobtained by long-time monitoring to make a diagnosis are anticipated.Not only utilization of remote biological information sensing at medicalfacilities but also utilization thereof at home to prevent sudden deathduring sleep and to monitor patients having sleep apnea syndrome areanticipated. Further, an application in which a change in the physicalcondition is constantly monitored by constantly obtaining physicalinformation data in daily life at home or at work and analyzing dataaccumulated in a server via a cloud network and the analysis result isused for health management and an application in which the obtained datais shared among medical facilities and is used for medical treatment arealso anticipated. To constantly obtain such biological information,systems that constantly obtain biological information without causing asubject to feel constrained and without being noticed by the subject aredesired. Systems using cameras enable non-constrained remote measurementand thus are considered to be optimum for such applications.

However, the privacy needs to be considered when biological informationsensing is performed by using cameras in daily life. Since systems inwhich high-resolution images that enable identification of individualsimaged with cameras are stored in a storage device involve a risk ofimage leakage, such systems need to be avoided. Even in the case whereobtained images are not stored, systems in which cameras (or cameralenses) are seen from a measurement system can make the subjectpsychologically uncomfortable. Accordingly, systems with hidden camerasare desirable.

Systems for implementing remote biological information sensing by usingcameras have been developed by various research facilities in responseto the above-described strong demand in the medical and healthcarefields, and some products are now commercially available. The biggestchallenge of remote biological information sensing systems of therelated art that use cameras lies in the accuracy and stability ofmeasurement. In the case where images of the human body are capturedwith a camera, most of the light incident on the camera is lightreflected from the skin surface or a portion close to the skin surface.Since there are no blood vessels and no metabolism occurs at the stratumcorneum, which is the outermost layer of the skin, no biologicalinformation is obtained from surface reflected components. It isnecessary to detect light that has reached the inner portion of the skinand has been reflected from an epidermal portion where blood vessels arepresent. The component reflected from the skin surface is dominant inlight reflected from the skin, and light that has reached the innerportion of the skin is rapidly lost by strong light absorption of aliving body. Accordingly, the ratio of light containing biologicalinformation to the reflected light is low. Further, systems that do notinclude an illumination system for measurement and that capture imagesby using ambient light has an issue of instability that is caused by afluctuation in an obtained image signal due to a fluctuation in ambientlight.

Further, in the case of remote sensing, the instability of measurementdue to a body motion is a big issue. Since an obtained signal varies dueto a change in a measurement region and a change in direction (angle)toward the camera in response to body motion, stable measurement isdifficult to perform. As described before, most of the signal obtainedwith a camera is a component resulting from reflection from the skinsurface which does not contain biological information, and a signalcomponent containing biological information is weak. Since thereflection from the surface greatly changes because of changes in themeasurement region and in the direction due to a body motion, weakbiological information is not successfully obtained. This is the biggestchallenge of remote biological information sensing using cameras. It isnecessary to perform measurement in a stationary state by holding thebody still, and a benefit of not being constrained is not fullyprovided.

Since images of a subject are obtained in remote biological informationsensing using cameras, methods for reducing the influence of a bodymotion by using these images are used in some cases. In such methods, aface portion is detected by using a face recognition function fromimages obtained with a camera, a portion subjected to measurement isfurther recognized by performing face part recognition, and biologicalinformation is detected by constantly capturing an image of the portionsubjected to measurement even if there is a body motion. For example, ifa forehead portion is successfully detected by face part recognition,information relating to the forehead portion is successfully obtainedconstantly even if the forehead portion moves in the images due to abody motion.

However, the methods using image recognition have two issues. One isthat computation load is heavy because face parts are recognized byperforming feature extraction on the entire image. Thus, it is necessaryto perform fast image processing by using an expensive high-performancearithmetic unit or it is necessary to reduce frame rate of an imagecapturing device so that the next frame can be processed afterprocessing of one frame is finished. Fast processing is costly andresults in a larger and more expensive device. When slow processing isemployed, the measurement accuracy decreases. Another issue is that evenif the influence of a body motion is reduced by performing imagerecognition, the increase in detection accuracy is limited due to achange in the direction of a subject portion (angle of direction of anormal to the surface of the subject portion with respect to the frontaldirection of the camera) in response to a body motion. The reflectanceof surface reflected light is angle-dependent. Accordingly, an amount ofsurface reflected light that reaches the camera varies when thedirection of the measurement-target portion changes due to a bodymotion. Consequently, the detection accuracy decreases.

As described above, the biggest challenge of remote biologicalinformation sensing performed using cameras is the instability ofmeasurement due to a body motion. Because of low reliability resultingfrom this instability, remote biological information sensing usingcameras has not been used for various applications.

As described above, various attempts have been made to obtain basicbiological information for determining the health condition of a personthrough measurement. For example, a method for detecting heart rate in anon-contact manner on the basis of image information of a face or thelike obtained with a camera has been proposed in Japanese UnexaminedPatent Application Publication No. 2005-218507. In the method accordingto Japanese Unexamined Patent Application Publication No. 2005-218507,heart rate is determined by analyzing a spatial frequency component ofan obtained color image. However, since the accuracy achieved by thismethod decreases due to the influence of disturbance light, such aslight illuminating a room, stable detection is difficult.

Pulse oximeters are commonly used to measure blood oxygen saturation.Pulse oximeters radiate two wavelengths of light in a red tonear-infrared wavelength range onto a finger inserted therein andmeasure transmittance of the light. In this way, pulse oximeters arecapable of determining a ratio between an oxyhemoglobin concentrationand a deoxyhemoglobin concentration in blood. Pulse oximeters arecapable of measuring blood oxygen saturation with a simpleconfiguration. However, since pulse oximeters are contact-type devices,they may make people feel restrained.

Japanese Unexamined Patent Application Publication (Translation of PCTapplication) No. 2003-517342 discloses an example of a non-contact-typeblood oxygen saturation measuring device. This device measures, by usinga white light source and a laser light source, blood oxygen saturationon the basis of a laser Doppler effect of laser light scattered behindthe surface of a living body. This method, however, makes aconfiguration of the device complex, and the resulting signal is weak.

In Japanese Unexamined Patent Application Publication (Translation ofPCT application) No. 2014-527863, a method for measuring, by using anordinary color camera, blood oxygen saturation while removing theinfluence of ambient light has been proposed. Since this method isgreatly influenced by reflected light reflected from the surface ofskin, it is difficult to measure blood oxygen saturation stably at ahigh accuracy.

As described above, the non-contact-type measurement methods formeasuring biological information, such as heart rate, blood pressure, orblood oxygen saturation of the related art have issues related to theaccuracy and the stability.

To measure biological information by using a camera, it is necessary tolocate a measurement-target region (e.g., a forehead region) in imagesobtained with a camera and to detect biological information by usingimage information in the region. Types of the method for locating themeasurement-target region include a method in which themeasurement-target region is specified before measurement and a methodin which the measurement-target region is automatically set based onimages. In the method in which the measurement-target region isspecified before measurement, a person who carries out the measurementspecifies the measurement-target region based on an image of a subjectbefore starting the measurement and continuously performs measurement atthe same portion during the measurement. This method is simple but doesnot allow the subject to move during the measurement. Accordingly, abenefit of non-contact measurement, which is not being constrained, islost. To avoid this, the method in which the measurement-target regionis automatically set is sometimes used. In this method, for example,when the measurement-target region is a forehead region, the cameraperforms face recognition on each obtained image and further performsface part recognition to locate the forehead portion in the image. Then,measurement can be performed at that portion.

FIG. 21 is a diagram schematically illustrating an example (comparativeexample) of such a system. An image capturing device 2, which is acamera of this system, includes a camera casing 6 including an imagesensor 7, and an optical system 5 including lenses. The image sensor 7of the image capturing device 2 includes or is connected to anarithmetic unit (or arithmetic circuit). The arithmetic unit performsface recognition on each obtained image (e.g., part (a) of FIG. 21 ) andextracts pieces of image data of a forehead portion (e.g., part (b) ofFIG. 21 ) after locating the forehead portion. The arithmetic unit thengenerates biological information, such as a change in heartbeat (e.g.,part (c) of FIG. 21 ), from the pieces of image data of the foreheadportion. Part (c) of FIG. 21 illustrates a temporal change in theaverage of the pieces of image data of the forehead portion illustratedin part (b) of FIG. 21 in the region of the forehead portion. A facepart recognition algorithm used in this system imposes heavy load on thecomputer performing the image processing. Accordingly, the cost of thearithmetic unit increases to implement fast processing. In addition, themethod using image recognition has an issue in that the recognitionaccuracy decreases when the orientation of the body changes or when partof the face is hidden. The method using image recognition further has anissue in that it is easily affected by ambient light. For these reasons,it is difficult to continuously perform stable measurement.

In addition to the issues described above, the method using facerecognition also has an issue in that measurement is not successfullyperformed at a portion (such as an arm or chest) other than the face.Further, it involves an issue regarding consideration for the privacy.The subject is psychologically stressed out by constant image capturingwith a camera. However, highly accurate image recognition requires theuse of a high-resolution camera for image capturing. Accordingly,constant image capturing with a camera may impose psychological load onthe subject.

The inventor has focused on the above-described issues and has studied aconfiguration for addressing the issues. The inventor consequently hasfound out that the issues can be addressed by obtaining an image byusing a light source that projects a light dot pattern onto the surfaceof a living body, by detecting a living body region (e.g., a human bodyregion) in the image on the basis of a ratio between a componentrelating to directly reflected light (also referred to as “surfacereflected light”) and a component relating to scattered light scatteredinside the living body (referred to as “inside body scattered light”) inthe image, and by separating, through signal processing, the componentrelating to the directly reflected light from the component relating tothe inside body scattered light in the detected living body region. Thatis, a biological information detection device first detects a regionthat is estimated to be a living body in an image and obtains biologicalinformation in that region. Such a method can greatly reduce an amountof computation relating to image processing and enables fast and stablebiological information detection as described in detail below.

A biological information detection device according to an aspect of thepresent disclosure includes a first light source, an image capturingdevice, and one or more arithmetic circuits. The first light source, inoperation, projects first dots onto a target including a living body.The first dots are formed by first light. The image capturing device, inoperation, generates and outputs a first image signal representing afirst image of the target onto which the first dots are projected. Theimage capturing device includes first photodetector cells that, inoperation, detect second light returning from the target onto which thefirst dots are projected. The one or more arithmetic circuits, inoperation, detect a first portion corresponding to at least a part ofthe living body in the first image by using the first image signal andcalculate biological information of the living body by using imagesignal of the first portion.

The first arithmetic circuit successfully detects a living body regionon the basis of a ratio between a signal of pixels in a region ontowhich the dot pattern is projected and a signal of pixels in an adjacentregion of the region. For example, the first arithmetic circuit iscapable of determining whether a living body is present at a positioncorresponding to a specific pixel of the image on the basis of a ratio(referred to as contrast) between a standard deviation of a pixel valueof the specific pixel and pixel values of a plurality of pixels locatedadjacent to the specific pixel and an average of the pixel value of thespecific pixel and the pixel values of the plurality of pixels locatedadjacent to the specific pixel. The second arithmetic circuit generatesand outputs information concerning the living body by mainly using asignal relating the region in which the dot pattern is not projected outof the image signal. With such a configuration, biological informationis successfully obtained at a high accuracy.

Herein, the term “biological information” refers to various kinds ofinformation relating to a living body, such as heart rate, blood flow,blood pressure, blood oxygen saturation, and respiratory information.Herein, the term “biological information” also refers to informationindicating the state of a person, such as the concentration degree or anemotion of a person, which is determined from these pieces ofinformation.

Principle

A principle allowing a biological information detection device to obtainhighly accurate biological information will be described below.

Biological information detection devices according to embodiments of thepresent disclosure use light having a wavelength in a range fromapproximately 650 nm to approximately 950 nm. This wavelength range iswithin a wavelength range of red to near-infrared light. Herein, theterm “light” is used not only for visible light but also for infrared.The above wavelength range is called “optical tissue window” and isknown as a range in which absorbance in the body is low.

FIG. 22 is a diagram illustrating wavelength dependency of a lightabsorption coefficient and an inside-body light scattering coefficientfor each of oxyhemoglobin, deoxyhemoglobin, melanin, and water. Light ina visible light range of 650 nm or shorter is absorbed mainly by blood(i.e. hemoglobin), and light in a wavelength range longer than 950 nm isabsorbed mainly by water. Therefore, light in these wavelength ranges isnot suitable for obtaining biological information. In contrast, in awavelength range from approximately 650 nm to approximately 950 nm, theabsorption coefficients for hemoglobin and water are relatively low, andthe scattering coefficient is relatively high. The scatteringcoefficient is larger than the absorption coefficient by an order ofmagnitude or more, and scattering is dominant in the wavelength band of“optical tissue window” in terms of interaction between the skin andnear-infrared light. Therefore, light of this wavelength range returnsto the body surface after entering the body and being stronglyscattered. Since such optical characteristics are unique to the skin,the use of these scattering-reflection characteristics allows the humanbody to be distinguished from other substances.

Biological information detection devices according to embodiments of thepresent disclosure mainly utilize light of this wavelength rangecorresponding to the “optical tissue window”. Since the use of a dotarray light source, for example, enables spatial separation anddetection of light directly reflected from the living-body surface andreturning light that has been scattered inside the living body,biological information can be efficiently obtained.

FIG. 1A is a diagram illustrating a schematic configuration of abiological information detection device according to an illustrativeembodiment of the present disclosure. The biological informationdetection device includes a light source 1 and an image capturing device2, which is a camera. The light source 1 is an array point light sourcethat projects a plurality of discretely arranged points (also referredto as “arrayed points” or “dot pattern” herein) onto a target includinga living body 3. The light source 1 is arranged such that a plurality ofpoints are projected onto the living body 3. The image capturing device2 includes an image sensor (also referred to as an “imaging element”),captures an image of a living-body surface 4, and generates and outputsan image signal.

FIG. 1B is a diagram for explaining characteristics of the image of theliving-body surface 4, obtained by the image capturing device 2.Outgoing light L0 from the light source 1 is reflected by theliving-body surface 4. Surface reflected light L1 reflected by theliving-body surface 4 maintains an image of the arrayed points formed bythe light source 1. In contrast, inside-body scattered light L2 thatexits from the living-body surface 4 after entering the living body 3and being scattered inside the living body 3 no longer maintains theimage of the arrayed points formed by the light source 1 because ofstrong scattering that occurs inside the living body 3. The use of thelight source 1 allows the surface reflected light L1 and the inside-bodyscattered light L2 to be spatially separated from each other easily.

The living body 3 illustrated in FIG. 1A represents human skin andincludes epidermis 33, dermis 34, and a subcutaneous tissue 35. No bloodvessels are present at the epidermis 33, whereas capillaries 31 andarterioles/venules 32 are located at the dermis 34. Since there are noblood vessels at the epidermis 33, the surface reflected light L1 doesnot contain information relating to blood. Since the epidermis 33includes melanin that strongly absorbs light, the surface reflectedlight L1 reflected from the epidermis 33 becomes noise whenblood-related information is obtained. Thus, the surface reflected lightL1 is not only useless to obtain blood-related information but alsodisturbs acquisition of accurate blood-related information. To detectbiological information at a high accuracy, it is extremely important tosuppress the influence of the surface reflected light and to efficientlyobtain information of the inside-body scattered light.

To address the issues described above, embodiments of the presentdisclosure have a novel configuration with which directly reflectedlight and inside-body scattered light are spatially separated by using alight source that projects arrayed points onto a living body and animage capturing device (or an image capturing system). With this novelconfiguration, information concerning the living body can be measured ata high accuracy in a non-contact manner.

In the related art, methods using polarizing illumination such as theone disclosed in Japanese Unexamined Patent Application Publication No.2002-200050 have been used to separate directly reflected lightreflected from the living-body surface. In such methods using polarizingillumination, a polarizer having a polarized light transmission axisperpendicular to a polarization direction of illuminating lightreflected from an image-capturing target is used. The influence ofsurface reflected light can be suppressed by capturing an image with acamera through such a polarizer. However, since the degree ofpolarization of surface reflected light reflected from an uneven surfacesuch as skin changes depending on the position, separation of suchdirectly reflected light is not sufficient. With a method according toembodiments of the present disclosure, the influence of surfacereflected light can be suppressed more effectively because directlyreflected light and scattered light are successfully spatiallyseparated.

The present disclosure includes, for example, aspects recited in thefollowing items.

-   -   [Item 1] In accordance with Item 1 of the present disclosure, a        biological information detection device includes a first light        source, an image capturing device, and one or more arithmetic        circuits. The first light source, in operation, projects first        dots onto a target including a living body. The first dots are        formed by first light. The image capturing device, in operation,        generates and outputs a first image signal representing a first        image of the target onto which the first dots are projected. The        image capturing device includes first photodetector cells that,        in operation, detect second light returning from the target onto        which the first dots are projected. The one or more arithmetic        circuits, in operation, detect a first portion corresponding to        at least a part of the living body in the first image by using        the first image signal and calculate biological information of        the living body by using image signal of the first portion.    -   [Item 2] In the biological information detection device        according to Item 1,    -   the second light may include third light from a first position        on a surface of the target and fourth light from a second        position on the surface of the target, the first position being        a position onto which at least one first dot among the first        dots is projected, and the second position being a position that        is different from any of positions onto which the first dots are        projected and that is located around the first position,    -   the one or more arithmetic circuits may, in operation, detect        the first portion in the first image by using a first image        signal component corresponding to the third light and a second        image signal component corresponding to the fourth light, the        first image signal component and the second image signal        component being contained in the first image signal, and    -   the one or more arithmetic circuits may, in operation, calculate        the biological information of the living body by using image        signal of the second position in the first portion.    -   [Item 3] In the biological information detection device        according to Item 2,    -   the one or more arithmetic circuits may, in operation, determine        a ratio between intensity of the first image signal component        and intensity of the second image signal component and may, in        operation, detect the first portion of the first image by using        the ratio.    -   [Item 4] In the biological information detection device        according to Item 2,    -   the one or more arithmetic circuits may, in operation, detect        the first portion of the first image by using ratios between a        standard deviation and an average of intensity of the first        image signal component and a standard deviation and an average        of intensity of the second image signal component.    -   [Item 5] In the biological information detection device        according to any one of Items 1 to 4,    -   the first light may include light having a wavelength that is        greater than or equal to 650 nm and less than or equal to 950        nm.    -   [Item 6] In the biological information detection device        according to any one of Items 1 to 5,    -   the biological information may include at least one piece of        information selected from the group consisting of heart rate of        the living body, blood pressure of the living body, blood flow        of the living body, blood oxygen saturation of the living body,        melanin concentration at skin of the living body, presence or        absence of a spot at skin of the living body, and presence or        absence of a bruise at skin of the living body.    -   [Item 7] In the biological information detection device        according to any one of Items 1 to 6,    -   the image capturing device may further include    -   a first bandpass filter that passes the second light, and    -   an image sensor having an imaging surface on which the first        photodetector cells are disposed and on which light that has        passed through the first bandpass filter is incident.    -   [Item 8] In the biological information detection device        according to any one of Items 1 to 7,    -   the one or more arithmetic circuits may, in operation,        calculate, as the biological information, at least one piece of        information selected from the group consisting of heart rate of        the living body, blood pressure of the living body, and blood        flow of the living body by using a temporal change in a value        obtained by performing lowpass filtering processing on at least        part of the image signal of the first portion.    -   [Item 9] The biological information detection device according        to any one of Items 1 to 8, may further include    -   a second light source that, in operation, projects second dots        onto the target, the second dots being formed by fifth light,        wherein:    -   the first light may include light having a waveform that is        greater than or equal to 650 nm and less than or equal to 800        nm,    -   the fifth light may include light having a waveform that is        greater than or equal to 800 nm and less than or equal to 950        nm,    -   the image capturing device may further include second        photodetector cells that, in operation, detect sixth light from        the target onto which the second dots are projected, and    -   the image capturing device may, in operation, generate and        output a second image signal representing a second image of the        target onto which the second dots are projected.    -   [Item 10] In the biological information detection device        according to Item 9,    -   the image capturing device may further include    -   an image sensor having an imaging surface that is divided into a        first region in which the first photodetector cells are disposed        and a second region in which the second photodetector cells are        disposed,    -   a first optical system that, in operation, forms the first image        in the first region, and    -   a second optical system that, in operation, forms the second        image in the second region.    -   [Item 11] In the biological information detection device        according to Item 10,    -   the image capturing device may further include    -   a first bandpass filter that is disposed in a path of the second        light and that passes the second light, and    -   a second bandpass filter that is disposed in a path of the sixth        light and that passes the sixth light.    -   [Item 12] In the biological information detection device        according to Item 9,    -   the image capturing device may further include    -   an image sensor having an imaging surface on which the first        photodetector cells and the second photodetector cells are        disposed, the image sensor including        -   first bandpass filters that face the respective first            photodetector cells and that pass the second light, and        -   second bandpass filters that face the respective second            photodetector cells and that pass the sixth light, and    -   an optical system that, in operation, forms the first image and        the second image on the imaging surface.    -   [Item 13] In the biological information detection device        according to Item 9,    -   the image capturing device may further include        -   an image sensor having an imaging surface on which the first            photodetector cells, the second photodetector cells, and            third photodetector cells are disposed, the image sensor            including            -   first bandpass filters that face the respective first                photodetector cells and that pass the second light,            -   second bandpass filters that face the respective second                photodetector cells and that pass the sixth light, and            -   third bandpass filters that face the respective third                photodetector cells and that pass visible light, and        -   an optical system that, in operation, forms the first image            and the second image on the imaging surface, wherein:    -   the third bandpass filters may include color filters having        different transmitting wavelength ranges, and    -   the image sensor may, in operation, generate and output a color        image signal by using the third photodetector cells.    -   [Item 14] In the biological information detection device        according to any one of Items 9 to 13,    -   the one or more arithmetic circuits may, in operation, calculate        information representing blood oxygen saturation of the living        body by using the first image signal and the second image        signal.    -   [Item 15] In the biological information detection device        according to any one of Items 9 to 13,    -   the one or more arithmetic circuits may, in operation,    -   calculate blood flow of the living body and blood oxygen        saturation of the living body by using the first image signal        and the second image signal, and    -   generate information representing at least one state selected        from the group consisting of a physical condition of the living        body, an emotion of the living body, and a degree of        concentration of the living body by using the blood flow of the        living body and the blood oxygen saturation of the living body.    -   [Item 16] In the biological information detection device        according to any one of Items 9 to 13,    -   in a case where the first image and the second image include at        least one target portion selected from the group consisting of a        forehead of the living body and a nose of the living body,    -   the one or more arithmetic circuits may, in operation,        -   calculate a temporal change in blood flow and a temporal            change in blood oxygen saturation at the at least one target            portion by using the first image signal and the second image            signal, and        -   generate information representing at least one state            selected from the group consisting of a physical condition            of the living body, an emotion of the living body, and a            degree of concentration of the living body by using the            temporal change in blood flow and the temporal change in            blood oxygen saturation.    -   [Item 17] In the biological information detection device        according to any one of Items 9 to 13,    -   in a case where the first image and the second image include a        forehead and a nose of the living body,    -   the one or more arithmetic circuits may, in operation,        -   calculate a temporal change in blood flow and a temporal            change in blood oxygen saturation at the forehead and a            temporal change in blood flow and a temporal change in blood            oxygen saturation at the nose by using the first image            signal and the second image signal, and        -   generate information representing at least one state            selected from the group consisting of a physical condition            of the living body, an emotion of the living body, and a            degree of concentration of the living body, based on a first            comparison between the temporal change in blood flow at the            forehead and the temporal change in blood flow at the nose            and based on a second comparison between the temporal change            in blood oxygen saturation at the forehead and the temporal            change in blood oxygen saturation at the nose.    -   [Item 18] In the biological information detection device        according to any one of Items 1 to 17,    -   the first light source may be a laser light source.    -   [Item 19] In the biological information detection device        according to any one of Items 1 to 18,    -   the image capturing device may further include    -   an image sensor having an imaging surface on which the first        photodetector cells are disposed,    -   an optical system that, in operation, forms the first image on        the imaging surface, and    -   an adjusting mechanism that, in operation, adjusts focus of the        optical system to maximize contrast of the first image.    -   [Item 20] In the biological information detection device        according to any one of Items 1 to 19,    -   the one or more arithmetic circuits may, in operation,    -   determine whether the first image includes at least one target        portion selected from the group consisting of a forehead, a        nose, a mouth, an eyebrow, and hair of the living body by using        the first image signal, and    -   calculate the biological information of the living body in        response to the first arithmetic circuit determining that the        first image includes the at least one target portion.    -   [Item 21] In the biological information detection device        according to any one of Items 1 to 20,    -   the one or more arithmetic circuits may, in operation, further        calculate different biological information of the living body by        using image signal of a second portion of the first image, the        second portion being different from the first portion.    -   [Item 22] In the biological information detection device        according to any one of Items 1 to 20,    -   the one or more arithmetic circuits may, in operation, further        compare a position of the first portion in the first image at a        first time point with a position of the first portion in the        first image at a second time point and may, in operation,        determine whether the living body has moved.

In the present disclosure, all or a part of any of a circuit, a unit, adevice, a member, or a portion; or all or a part of functional blocks inthe block diagrams may be implemented as one or more of electroniccircuits including, but not limited to, a semiconductor device, asemiconductor integrated circuit (IC), or a large scale integration(LSI). The LSI or IC can be integrated into one chip, or also can be acombination of a plurality of chips. For example, functional blocksother than a memory may be integrated into one chip. The name used hereis LSI or IC, but it may also be called system LSI, very large scaleintegration (VLSI), or ultra large scale integration (ULSI) depending onthe degree of integration. A field programmable gate array (FPGA) thatcan be programmed after manufacturing an LSI or a reconfigurable logicdevice that allows reconfiguration of the connection or setup of circuitcells inside the LSI can be used for the same purpose.

Further, it is also possible that all or a part of the functions oroperations of the circuit, the unit, the device, the member, or theportion are implemented by software-based processing. In such a case,the software is recorded on one or more non-transitory recording mediasuch as a read-only memory (ROM), an optical disc, or a hard disk drive,and when the software is executed by a processor, the software causesthe processor together with peripheral devices to execute the functionsspecified in the software. A system or a device may include such one ormore non-transitory recording media on which the software is stored anda processor together with necessary hardware devices such as aninterface.

Embodiments of the present disclosure will be described in more detailbelow. The following embodiments relate mainly to a biologicalinformation detection device that measures biological information in anon-contact manner, assuming that a face of a person is a living-bodysurface. Note that techniques of the embodiments of the presentdisclosure are applicable not only to a face of a person but also toportions other than the face of a person or to skin of animals otherthan the human.

First Embodiment

A system in which a technique of embodiments of the present disclosureis applied to non-contact heartbeat measurement will be described as afirst embodiment. With a growing interest in healthcare, importance ofconstant biological information sensing is increasing. A system capableof constantly obtaining biological information in a non-contact mannerthrough measurement is essential not only at hospitals but also forhealth management in daily life. The system according to the firstembodiment is capable of monitoring heartbeat and a change in heart ratein a non-contact manner.

FIG. 2 is a diagram illustrating a schematic configuration of a livingbody detection system according to the first embodiment. As illustratedin FIG. 2 , the living body detection system according to the firstembodiment includes a light source 1, an image capturing device 2, and acomputer 20. The light source 1 is located at a position apart from aliving body 3 and emits a light beam of a near-infrared wavelengthrange. The image capturing device 2 is a camera capable of recordingimages of a living-body surface 4 illuminated with light. The computer20 is connected to the light source 1 and the image capturing device 2.The computer 20 is capable of separating and measuring a componentrelating to surface reflected light L1 reflected from the living-bodysurface 4 and a component relating to inside-body scattered light L2from a captured image. The computer 20 is capable of detecting whetheran image includes a region corresponding to a living body on the basisof intensity of the surface reflected light L1 and intensity of theinside-body scattered light L2. The computer 20 is also capable ofcalculating and outputting biological information, such as heart rate,from a signal in a region of the image corresponding to a living body.

The light source 1 is designed to project a dot pattern onto theliving-body surface 4. Typically, a dot pattern is a collection oftwo-dimensionally arranged small bright spots. A dot pattern ofone-dimensionally arranged bright spots may be used depending on theapplication. In the first embodiment, for example, a random dot patternlaser projector RPP017ES available from Osela Inc. in Canada is usableas the light source 1. This laser light source emits a near-infraredlaser beam of 830 nm and projects a laser dot pattern including 57446spots in a 45°×45° viewing angle.

FIG. 3A is a diagram illustrating an example of a configuration of theimage capturing device 2 and examples of a generated image andbiological information. The image capturing device 2, which is a camera,includes an optical system 5 and a camera casing 6. The optical system 5may be a set of a plurality of lenses. The camera casing 6 includes animage sensor 7 and a bandpass filter 8. The bandpass filter 8 passesonly light having a wavelength of 830 nm±10 nm, which is the wavelengthof light emitted from the light source 1.

In the case where the subject is a person, the image sensor 7 obtains asignal of an image including a plurality of points each having abrightness corresponding to an infrared reflectance at a correspondingposition. Part (a) of FIG. 3A illustrates an example of an imagerepresented by such an image signal. A arithmetic circuit of thecomputer 20 detects, through signal processing, only a regioncorresponding to a human body as illustrated in part (b) of FIG. 3A.This detection is performed based on a ratio between a signal componentrelating to the surface reflected light L1 and a signal componentrelating to the inside-body scattered light L2.

As described before, a living body has a specific optical propertycalled “optical tissue window” for a wavelength range of red tonear-infrared light. Since human skin has a small absorption coefficientand a large scattering coefficient in this wavelength range, light thathas transmitted through the skin surface, which is the living-bodysurface 4, repeats multiple scattering and scatters inside the body andthen exits from a wide area of the living-body surface 4. Accordingly,as illustrated in part (c) of FIG. 3A as an enlarged view, regions basedon the inside-body scattered light L2 are present around the respectivebright spots based on the surface reflected light L1 in the regioncorresponding to the human body in the image. The living bodycharacteristically has a high proportion of scattered light relative tosurface reflected light in the above wavelength range. In contrast,objects other than the living body have a very high proportion ofsurface reflected light relative to scattered light. Accordingly, aregion corresponding to a living body can be detected based on the ratiobetween directly reflected light and scattered light. Further,biological information is successfully obtained fast by using signals ofa plurality of pixels included in the living body region of the obtainedimage. Human body region detection based on the optical characteristicsof the skin according to the first embodiment is fast and highlyaccurate, compared with image-recognition-based methods of the relatedart. Detection of a human body region and highly accurate humandetection performed using the resulting information implement fast andhighly accurate biological information sensing.

FIG. 3B is a block diagram illustrating a configuration of the computer20. The computer 20 includes an input interface (IF) 21, a firstarithmetic circuit 22, a second arithmetic circuit 23, a memory 25, acontrol circuit 26, an output interface (IF) 24, and a display 27. Theinput IF 21 is electrically connected to the image capturing device 2.The first arithmetic circuit 22 performs signal processing for detectinga region corresponding to a living body in an image. The secondarithmetic circuit 23 calculates biological information (pulses in thefirst embodiment) by using image data of the detected human body region.The memory 25 stores various kinds of data. The control circuit 26controls operations of the entire device. The output IF 24 outputs datagenerated by the second arithmetic circuit 23. The display 27 displays aprocessing result. The first arithmetic circuit 22 and the secondarithmetic circuit 23 may each be an image processing circuit, forexample, a digital signal processor (DSP). FIG. 3B illustrates the firstarithmetic circuit 22 and the second arithmetic circuit 23 as differentblocks; however, they may be implemented as a single circuit. Thecontrol circuit 26 may be an integrated circuit, for example, a centralprocessing unit (CPU) or a microcomputer. The control circuit 26 runs acontrol program stored, for example, in the memory 25 to performcontrol, such as providing an instruction to switch on to the lightsource 1, an instruction to capture an image to the image capturingdevice 2, and an instruction to perform computation to the firstarithmetic circuit 22 and the second arithmetic circuit 23. The controlcircuit 26, the first arithmetic circuit 22, and the second arithmeticcircuit 23 may be implemented as a single integrated circuit. In theexample illustrated in FIG. 3B, the computer 20 includes the display 27;however, the display 27 may be an external device electrically connectedto the computer 20 wirelessly or by a cable. The computer 20 may obtain,via a communication circuit (not illustrated), image information fromthe image capturing device 2 located at a remote place.

In the example illustrated in FIG. 3A, the second arithmetic circuit 23averages signal components based on the inside-body scattered light L2of the human body region detected by the first arithmetic circuit 22.Averaging is performed for each frame of a moving image, for example.Consequently, data representing a temporal change in the average of thesignal components based on the inside-body scattered light L2 isobtained as illustrated in part (d) of FIG. 3A. Heart rate (the numberof heartbeats per unit time) can be determined by determining a periodor a frequency from this data.

Further, respiration measurement is also carried out simultaneously withheart rate measurement by using the similar system configuration. FIG.3C is a diagram schematically illustrating such a system that performsrespiration sensing. The hardware of this example is the same as thatillustrated in FIG. 3A. With this configuration, non-contact respirationmonitoring can be implemented through image signal processing.

The human respiratory interval is about 3 to 4 seconds (15 to 20times/minute), and the chest portion and the abdominal wall expand andcontract due to respiration by about 5 mm in the case of adults. If thismotion of the chest portion is successfully measured with an imagecapturing device, respiration is successfully monitored.

A method for monitoring respiration on the basis of an amount of shiftof the chest portion by using a near-infrared dot array light source isdisclosed in, for example, Aoki and two others, “Non-contact andUnrestrained Respiration Watch System for Sleeping Person UsingNear-infrared Bright Spots Matrix Irradiation”, IEEJ Transactions onElectrical and Electronic Engineering, C. Electronics, Information andSystems, Jun. 1, 2004, Vol. 124(6), pp. 1251-1258 (hereinafter, referredto as NPL 1). The system described in NPL 1 implements highly accuratenon-contact respiration sensing by determining an examination-targetregion for a stationary subject in advance. The system described in NPL1 assumes non-contact respiration monitoring during sleep, and onlyrespiration is monitored by using a large system.

In contrast, the use of the system according to the first embodimentillustrated in FIG. 3C enables respiration sensing to be performedsimultaneously with heart rate sensing by using a small and inexpensivedevice. Further, the use of the system according to the first embodimentillustrated in FIG. 3C enables stable measurement in accordance with abody motion of the subject.

Referring to FIG. 3C, a respiration measurement method according to thefirst embodiment will be described below. The first arithmetic circuit22 performs human body detection on a near-infrared image (part (a) ofFIG. 3C) obtained by the image sensor 7 by using the above-describedmethod and estimates a face region from data of the human body region(part (b) of FIG. 3C). The first arithmetic circuit 22 further estimatesthe position of the chest on the basis of the center of the face region.For example, a position that is shifted below from the center of theface region by a distance that is 1 to 1.2 times as large as thevertical length of the face region is estimated to be a chest region.Then, a temporal change in the position of the dot array in this chestregion (part (d) of FIG. 3C) is measured. At that time, a variation inthe position less than the pixel pitch can be measured by averagingvariations in position of a plurality of dots. Note that positions ofthe dots in the captured image barely change in response tolateral-direction motion of the target (chest) and that the positions ofthe dots change only in response to depth-direction motion of the targetand such a change is measured. In methods using an ordinary image, bothlateral-direction motion of the target and depth-direction motion of thetarget are detected as movement on pixels. Since lateral-directionmotion is detected more sensitively, measurement accuracy is low. Incontrast, in the first embodiment, lateral-direction motion of thetarget is no longer detected and only depth-direction motion of thetarget is detected with the use of the dot array light source.Accordingly, highly accurate respiration monitoring can be performed. Anamount of shift in the position of the dot array pattern in the chestregion between frames can be determined by calculating anautocorrelation of the dot array pattern. This average shift amountrepresents up-down motion of the chest due to respiration. By plottingthe average shift amount of the dot array pattern with respect to thetime axis as illustrated in part (e) of FIG. 3C, monitoring ofrespiration can be performed. With the configuration of the firstembodiment, highly accurate respiration sensing can be performed whiletracking the chest region even if there is a body motion.

In the example illustrated in FIG. 3C, a method for measuring heart rateby using data of pixels in a forehead region of images is the same asthe method illustrated in FIG. 3A. In the example of FIG. 3C, both heartrate and respiration are measured; however, respiration alone may bemeasured.

An example of the living body detection method that is carried out byusing actual data will be described below.

FIG. 4A illustrates an example of an image obtained by an ordinary imagecapturing device that detects visible light. The central part shows aface F of a person. A left diagram in FIG. 4B shows an image of the samescene as that of FIG. 4A captured with the image capturing device 2according to the first embodiment when the place is illuminated by thelight source 1 of a wavelength of 830 nm. In this image, it is difficultto recognize the face F due to strong reflection from a box B located inthe foreground. Accordingly, to detect a human body, the firstarithmetic circuit 22 calculates contrast between directly reflectedlight and scattered light from a near-infrared image.

FIG. 5 is a diagram illustrating an example of a pixel region used tocalculate contrast. Image data is stored as two-dimensional intensitydata in the memory 25. Here, Pij denotes data of a pixel located in ani-th column in the horizontal (x) direction of a j-th row in thevertical (y) direction. Contrast Cij for this pixel (i, j) is defined asfollows:

Cij=Sij/Aij.

Here, Sij and Aij respectively denote a standard deviation and anaverage of pieces of data of pixels in a 7×7 pixel region centered atthe pixel (i, j). Since the standard deviation Sij decreases as theratio of scattered light to directly reflected light increases, thevalue of the contrast Cij decreases. After repeatedly performing thisprocessing for all pixels, the first arithmetic circuit 22 extracts onlypixels for which the value of the contrast Cij is within a predeterminedrange. An example of an image that shows a part of a region where0.2<Cij<0.47 is the image on the right in FIG. 4B. In this image, pixelsfor which the value of the contrast Cij is within the above range areshown in white, and the rest of the pixels are shown in black. The imageindicates that a living body (i.e., the face F) is correctly extracted.

As described above, the first arithmetic circuit 22 according to thefirst embodiment calculates the contrast Cij, which is a ratio betweenthe standard deviation of pixel values of a specific pixel included inan image and a plurality of neighboring pixels of the specific pixel andthe average of the pixel values of the specific pixel and the pluralityof neighboring pixels. Based on the value of the contrast Cij, the firstarithmetic circuit 22 is able to determine whether a living body islocated at a position corresponding to the specific pixel and outputinformation indicating the presence or absence of the living body.

According to the first embodiment, a living body hidden behind manyobjects can be efficiently detected by utilizing an optical propertyspecific to a living body. The average and the standard deviation arederived for a 7×7 pixel region to derive contrast of the image (i.e.,contrast of directly reflected light and scattered light) in thisexample; however, this size of the pixel region is merely an example.The size (i.e., the number of pixels) of the pixel region used tocompute the contrast is appropriately set in accordance with the densityof a plurality of points formed by the light source 1 and the resolutionof the image capturing device 2. To suppress a variance in thecalculation result, a plurality of (e.g., three or more) points may beincluded in a pixel region subjected to computation. The accuracy of thecalculated contrast value improves by increasing the number of pixelsincluded in the region subjected to computation; however, the resolutionof the resulting image of the living body decreases. Accordingly, thenumber of pixels included in the region subjected to computation isappropriately set in accordance with the configuration and usage of thesystem. Further, not only the number of pixels subjected to computationbut also a pixel interval at which this processing is repeated alsoaffect the processing speed. In the above-described processing,computation is sequentially repeated for all the pixels; however, theprocessing speed can be increased by increasing the pixel intervalsubjected to computation although the resolution decreases. This pixelinterval can also be appropriately set in accordance with theconfiguration and usage of the system. Likewise, the predeterminedcontrast range is not limited to 0.2<Cij<0.47 and may be appropriatelyset in accordance with the configuration and usage of the system.

The first arithmetic circuit 22 detects, from a two-dimensional imagecaptured by the image capturing device 2, a region of the imagecorresponding to a human body by using the above method. The secondarithmetic circuit 23 then obtains biological information. Since thehuman body region is determined in the image by the first arithmeticcircuit 22, biological information is generated by using pixel data ofthis region. The second arithmetic circuit 23 generates, as biologicalinformation, data of temporal changes in heart rate and respiration asillustrated in FIG. 3C, for example. In this way, heart rate and achange in heartbeat can be monitored in a non-contact manner.

In biological information sensing systems using a camera according tothe related art, a method for detecting biological information byaveraging pieces of pixel data of a living body region of an image iscommonly used. In contrast, since the biological information sensingsystem according to the first embodiment uses a dot array light source,an unnecessary surface reflected light component reflected from the skinsurface can be removed from a two-dimensional image, and inside-bodyscattered light containing biological information can be selectivelyextracted. Images (surface reflected light) of a projected dot array aredetected as spots having large pixel values, and a component relating toscattering that occurs inside the body (inside-body scattered light) isdetected as regions that are located around the respective dots and havea smaller pixel value than the corresponding dots. Thus, by setting athreshold for light intensity and averaging pieces of pixel dataobtained by removing pieces of pixel data of a predetermined lightintensity or higher, inside-body scattered light can be efficientlyextracted. Through such processing, highly accurate biologicalinformation can be obtained.

FIG. 6 is a flowchart illustrating an example of an operation performedby the first arithmetic circuit 22 and the second arithmetic circuit 23according to the first embodiment. The operations of the firstarithmetic circuit 22 and the second arithmetic circuit 23 will bedescribed by using the case where a moving image is obtained by theimage sensor 7 as an example. The operation described below can beimplemented as a result of one or a plurality of processors executing acomputer program stored in a memory.

First, the first arithmetic circuit 22 extracts a region correspondingto a human body from the captured moving image (step S101). A method forextracting a human body region is as described above. Then, the secondarithmetic circuit 23 removes pieces of data for the central portions ofthe dot array, which correspond to directly reflected light components,from pieces of pixel value data of the extracted human body region byusing a threshold set in advance (step S102). Then, the secondarithmetic circuit 23 calculates an average of the pixel values(corresponding to inside-body scattered light components) of the humanbody region (step S103). Steps S101 to S103 described above areperformed for each frame of the moving image. The second arithmeticcircuit 23 calculates a period and an amplitude of a temporal change inthe average by using pieces of data of frames of a predetermined period(e.g., several to several tens of seconds) (step S104). In this way,information relating to blood flow in the body can be obtained. Sincearterial blood pumped out from the heart moves along the blood vesselswith fluctuation called pulses, the near-infrared absorptance andreflectance change in accordance with the pulse. Heart rate can bedetermined from a period of this fluctuation in the reflectance.Further, blood pressure and blood flow can be estimated from theamplitude of the pulse.

In the step of removing the directly reflected light components by usinga threshold set in advance (step S102), a constant threshold can be usedin the case where a distance between the image capturing device and thesubject is fixed. However, since the distance between the imagecapturing device and the subject often varies, it is usually desirablethat such a case be successfully coped with. Accordingly, for example,an average of pixel values of the entire living body region subjected tocomputation may be calculated, and the threshold may be changed inaccordance with the average. In such an embodiment, as the average ofpixel values becomes larger, the threshold can be set to be larger.

According to the first embodiment, for example, heart rate dataillustrated in part (d) of FIG. 3A can be obtained. Many methods formonitoring heartbeat by using an ordinary visible-light camera and anear-infrared camera in a non-contact manner have been proposed. Sinceseparation of surface reflected light components and scattered lightcomponents is insufficient in these methods of the related art,non-contact measurement is easily affected by disturbance light and itis difficult to implement stable and highly accurate measurement. Incontrast, in the first embodiment, stable and highly accurate heartbeatmeasurement can be performed by spatially separating surface reflectedlight components and scattered light components that are contained in anobtained image signal. For example, in remote heartbeat measurementusing a camera according to the related art, detection becomes unstabledue to a body motion in response to vocalization during a conversation,making it difficult to perform highly accurate heartbeat measurement.However, the use of the method according to the first embodiment enablesstable heartbeat measurement even when there is a body motion for aconversation.

According to the first embodiment, psychological stress of the subjectcan be estimated. It is known that psychological stress can be estimatedfrom a temporal fluctuation in heart rate. When the autonomic nervoussystem is functioning properly, the interval between heartbeatsfluctuates. It is known that the fluctuation in the interval betweenheartbeats decreases due to stress. The second arithmetic circuit 23according to the first embodiment is also capable of detecting whetherthe subject has psychological stress and detecting the degree ofpsychological stress on the basis of the change in the fluctuation inthe interval between heartbeats. To constantly perform stress sensing indaily life, an unconstrained and non-contact heartbeat sensingtechnology such as the first embodiment is important.

As described above, the use of the system according to the firstembodiment makes it possible to monitor heart rate or blood pressure allthe time including a sleeping period without constraining the subject.Consequently, for example, a system can be constructed that constantlymonitors the condition of the patient at hospitals and alerts medicalpersonnel when anything unusual occurs. In addition, for example,monitoring of heart rate of a patient who has sleep apnea syndrome canbe performed at nighttime at home. Further, since stress sensing can beeasily performed in daily life as described above, people can enjoytheir daily life more.

Second Embodiment

In the first embodiment, the system that detects a human body regionfrom an image and obtains biological information from the human bodyregion of the image has been described. Typical application examplesthat use human body detection will be described below as a secondembodiment. Development relating to human body detection is underway forthe purpose of detecting disaster victims buried under rubble or thelike at a disaster site, for example. Finding disaster victims within 72hours from occurrence of a disaster is critical in terms of the survivalrate of disaster victims. Accordingly, a simple and stable living bodydetection system is needed. The living body detection technology is alsoutilized in fields of security and transportation. The living bodydetection technology plays an important role to find an intruder in thefield of security and to detect foot passengers in the field oftransportation. There is also an increasing need for a system capable ofselectively detecting a living body (especially, person) in an imageincluding various constructions or objects. A living body can bedetected by the operation of the first arithmetic circuit 22 accordingto the first embodiment. However, more accurate and more reliable livingbody detection can be implemented by further performing sensing ofbiological information (e.g., presence or absence of pulses) on a regionwhere a living body is detected. Operations according to the secondembodiment will be described below for each application. The physicalconfiguration of the second embodiment is substantially the same as thatof the first embodiment.

(1) To Find Disaster Victims at Time of Disaster

Quickly finding disaster victims buried in rubble in response tooccurrence of a natural disaster, such as earthquake, tsunami, or debrisflow, is particularly important to save people's lives. There is “goldentime of 72 hours”, which indicates that the survival rate greatlydecreases after 3 days, and it is necessary to quickly find disastervictims in a chaos circumstance. The use of the system according to thesecond embodiment makes it possible to detect disaster victims hiddenbehind rubble in real time by capturing an image even in a circumstancewhere rubble is scattered everywhere. Since the system is small, thesystem can be installed on an unmanned aerial vehicle (UAV), which isso-called a drone, for example. This configuration makes it possible tocapture an image while remotely controlling the system at a remotelocation and to search for survivors even if a disaster site isdifficult to access because of a risk of a secondary disaster.

Human body detection can be performed through living body detectionprocessing performed by the first arithmetic circuit 22 according to thefirst embodiment. However, to improve the accuracy, the processingresult obtained by the second arithmetic circuit 23 is further utilizedin the second embodiment. Specifically, the second arithmetic circuit 23performs sensing of biological information (e.g., the presence orabsence of pulses) in a region (referred to as a living body region)estimated to correspond to a living body by the first arithmetic circuit22. In this way, a human body can be detected more accurately. Bydetermining the presence or absence of a body motion in the living bodyregion, erroneous detection can be reduced and reliability can beincreased. The presence or absence of a body motion can be determined onthe basis of whether there is a temporal change in the living bodyregion by comparing a plurality of consecutive frames, for example.Further, the living body detection accuracy can be increased drasticallyby determining the presence or absence of heartbeats by using pieces ofpixel data of the living body region. In accordance with the secondembodiment, fast and highly reliable living body detection can beperformed by using living body detection based on the opticalcharacteristics of the skin, body motion detection based shifting of theliving body region, and heartbeat measurement calculated from signalintensity in the living body region in combination. Further, since thephysical conditions of disaster victims can be determined frombiological information, the rescue priority can be determined based onthe data.

More reliable living body detection can be implemented in applicationsof human body detection described below by using pieces of informationobtained by (1) living body detection, (2) body motion detection, and(3) heart rate detection in combination in accordance with the secondembodiment.

(2) For Use in Monitoring

Surveillance cameras are widely used and contribute to safe and securedaily life of people. As the number of surveillance cameras increases,it becomes more important how and who checks a video image captured bythe surveillance cameras. Since it is difficult for a person to checkthe image all the time, a usage is common in which the image isaccumulated and the image is checked after occurrence of a problem(crime) to grasp the situation. A utilization method may be adopted inwhich the moment at which a problem occurs is captured from a real-timeimage and the problem is immediately dealt with. The use of thetechnique according to the second embodiment of the present disclosuremakes it possible to construct a system that recognizes a person whenthe person enters the field of view of a surveillance camera and warns aperson in charge to prompt the person in charge to check the image inreal time. A system can be constructed that frees the person in chargefrom the necessity of standing by in front of the monitor of thesurveillance camera and that displays a warning and the image on amobile terminal carried by the person in charge upon detecting a person.Such a system is suitable for monitoring at a backdoor of a warehouse orbuilding where people rarely appear or a place where access isrestricted. In addition, for a place, such as a building, where imagescaptured with many surveillance cameras are collectively monitored,highlighting a video image in which a certain person is detected may beuseful to prevent an unusual situation from being overlooked or to findout an unusual situation at an early stage.

More important information can be obtained for the purpose of monitoringby sensing biological information (presence or absence of pulses) byusing the second arithmetic circuit 23 in addition to detection of asuspicious person. The psychological nervousness can be estimated fromheart rate or the fluctuation in heartbeat of a suspicious person in amonitored image. The degree of danger (alert level) of that person canbe estimated from the estimated nervousness. Security systems thatdetect possible criminals from crowd at airports or commercialfacilities by using images obtained with cameras are under development.The living body detection/living body sensing system according to thesecond embodiment is also applicable for such a purpose.

As for monitoring, development of a method in which a computer performsobject recognition is in progress thanks to the advance in the imagerecognition technology, instead of a traditional method in which aperson judges a monitoring image. For such a usage, a common method isthat an image is transmitted to a host computer and the host computerperforms recognition. However, since this method requires image data betransmitted to the host computer, this method involves issues such as anincreasing amount of communications, a decreasing communication rate,and an increasing load of the host computer. If a surveillance camera iscapable of performing preliminary recognition and judgement on an image,the load for communication, storage, and computation can be greatlyreduced. However, if the recognition is not sufficiently reliable, therecognition may lead to overlooking of an event. Since a person can behighly reliably detected with the human body detection method accordingto the second embodiment, only a partial image including a person can beselectively transmitted to the host computer upon detecting the person.Consequently, the surveillance system can be efficiently operated.

In addition, the progress in the image recognition technology makes itpossible to identify an individual from an image at a high accuracy. Interms of identification of an individual from an image, a method inwhich an image is transmitted to a host computer and the host computerperforms recognition is commonly used; however, this method alsoinvolves issues relating to load for communication, storage, andcomputation as described above. An operation for extracting a faceportion for face recognition imposes a heavy load during computation.The use of the human body detection method according to the secondembodiment allows the face portion to be easily extracted from an image.Accordingly, only the part of the image for the face portion can betransmitted to the host computer for individual identification, and theload for identifying an individual can be greatly reduced. Further, ifthe number of people to be identified is limited, a surveillance camerais able to immediately identify an individual without using the hostcomputer by registering characteristics of the people in thesurveillance camera in advance.

(3) For Use in Vehicles

Installation of the system according to the second embodiment in avehicle makes it possible to recognize foot passengers on the street andimplement safer driving. Even when a person is hidden behind an objectand is scarcely seen, the system can detect the person and warn thedriver. In terms of automated driving, in a situation where a vehiclecannot stop in response to breaking and an accident is inevitable evenif the vehicle changes the direction to the left and to the right, aquestion about which direction the vehicle should head occurs. In such acase, it is effective to detect people with the system according to thesecond embodiment and to change the heading direction to a direction inwhich the vehicle can avoid people. Since it is desired that the systemquickly and highly accurately detect people in such a usage, the systemaccording to the second embodiment is particularly suitable.

(4) Person Detection Switch

There is a wide variety of usages in which power is switched on and offby detecting a person. For example, there are usages in which switchingof a device such as an air-conditioner or a light is controlled bydetecting a person in a room, an automatic door is controlled at a highaccuracy, a traffic light for foot passengers is controlled by detectingof a foot passenger at crossing, and brightness of an illumination of avending machine is changed. The second embodiment is applicable to suchusages. The use of the system according to the second embodiment canimplement a sophisticated switch that does not respond to an object orpet but responds only to people. In such a usage, a small persondetection sensor unit including the light source, the image capturingdevice, and the arithmetic circuit of the system according to the secondembodiment may be constructed.

(5) Biometric Authentication

Biometric authentication, such as fingerprint authentication, irisauthentication, and vein authentication, is widely used as a method forauthenticating an individual. With the increasing use of suchauthentication, cases and a risk of spoofing in biometric authenticationare increasing. An image duplication technology, such as copying, hasbeen used in image-based authentication. Recently, with an increasinguse of iris authentication and a three-dimensional printer, a risk ofspoofing using a highly precise duplicate is increasing. As acountermeasure for such a risk, a two-step authentication system iseffective. For example, a method is effective in which ordinarybiometric authentication is performed after checking that a target is aliving body by using the living body detection system according to thesecond embodiment. By checking that the target is a living body by usingthe living body detection system according to the second embodiment, thereliability of biometric authentication can be increased.

In the usages (1) to (4) out of the above-described usages, the secondarithmetic circuit 23 may generate image data in which a regiondetermined to be a human body is superimposed on a visible-light-basedimage and display the resulting image on a display. Since people have adifferent visual impression from a single image representing the humanbody region or from an image in which an infrared image and anear-infrared image representing the human body region are superimposed,there is an issue about how a person recognizes the position of a humanbody even when the human body is detected. To address this issue, avisible-light camera may be added to the system illustrated in FIG. 3A.The second arithmetic circuit 23 may superimpose a visible-light-basedimage obtained by the visible-light camera and a near-infrared imageobtained by the image sensor 7 to generate image data in which the humanbody region is superimposed on the visible-light-based image. Bydisplaying the human body region on the visible-light-based image in anemphasized manner, the noticeability can be increased. In usages inwhich a person makes a judgement after a human body is detected, asystem capable of superimposing an image of the human body region on avisible-light-based image is more effective.

Further, in the case where the system additionally including avisible-light camera is used, the second arithmetic circuit 22 mayextract an outline of an image from a visible-light-based image andremove a portion corresponding to the outline from a region estimated tobe a human body region. This is because an outline region of an objectis erroneously detected as a human body since the infrared reflectancesometimes changes greatly at the outline portion of an object. Byremoving the outline region, an image of a human body region containingless noise can be obtained.

In order to obtain a visible-light-based image and a near-infrared imagefor use in human body detection by using a single camera, avisible-light cut-off filter is removed from the camera and visiblelight and near-infrared light may be switched between every frame byassociating illumination light with the frame rate of the camera. Withsuch a configuration, a visible-light-based image and a near-infraredimage can be obtained by using a single camera. Advantages of thismethod are that there is no parallax between cameras and superimpositionof images is easy because a visible-light-based image and anear-infrared image can be obtained by using a single camera.

Third Embodiment

A more specific application example in which human body detection andbiological information sensing are used in combination will be describedas a third embodiment. As described above, the system according toembodiments of the present disclosure is capable of quickly detecting ahuman body and quickly and highly accurately obtaining biologicalinformation, such as heartbeats, from data of the detected human bodyregion. By using this system, a watch system installed in personalspaces such as a bathroom, a lavatory, and a bedroom can be implemented.It is particularly important to consider the privacy in personal spaces.Systems that constantly capture images of a subject with ahigh-resolution camera and that use such images involves a concern aboutviolation of privacy due to leakage of the images and imposespsychological load during image capturing due to the presence of thecamera. Such issues are coped with by the watch system according to thethird embodiment.

With population aging, it is said that ten thousands to twenty thousandsof people die in the bathroom in Japan. This number is much larger thanthe number of people who die in traffic accidents, which is four to fivethousands of people. The causes of death in in the bathroom includeaccidents (death by drowning) and diseases (attacks due tocardiovascular disorders and brain disorders). Many of people who die inthe bathroom are elderly people, and many people die in winter. Withpopulation aging, the annual number is also increasing. Death cases thatoccurred in the bathroom include many cases in which the life can besaved if an unusual situation can be detected soon after the occurrenceregardless of whether the death results from an accident or a disease.Since the bathroom is a closed private space, there are many cases wherelate discovery results in death. A system capable of watching a targetperson in the bathroom while paying considerations to privacy isstrongly desired.

FIG. 7A is a diagram schematically illustrating a biological informationdetection device according to the third embodiment and a processingprocess performed thereby. The configuration of the biologicalinformation detection device according to the third embodiment issubstantially the same as that illustrated in FIG. 3A. However, in thethird embodiment, the light source 1 and the image capturing device 2,which is a camera, are housed in a casing 15 having a water-resistancefunction in consideration for the use in the bathroom. The casing 15 hasan opening at a frontal surface thereof so as not to block light from alight source 1 and light returning from a subject 3. At the opening, afilter 16 that transmits near-infrared light is provided. Near-infraredlight emitted from the light source 1 passes through the filter 16 andis incident on the subject 3. The near-infrared light reflected from thesubject 3 passes through the filter 16 again, passes through an opticalsystem 5, which includes lenses of the image capturing device 2, and abandpass filter 8, and is incident on an image sensor 7.

The biological information detection device illustrated in FIG. 7Afurther includes a speaker 18 that issues an alert (alarm) and a controldevice 17. The control device 17 is connected to and controls the imagecapturing device 2, the light source 1, and the speaker 18. The controldevice 17 is a component equivalent to the computer 20 illustrated inFIG. 2 and includes the first arithmetic circuit 22, the secondarithmetic circuit 23, the memory 25, and the control circuit 26illustrated in FIG. 3B. The control circuit 26 of the control device 17instructs the speaker 18 to issue an alert upon detecting anythingunusual.

In the third embodiment, the system is configured to be water-proof andis configured such that the image capturing device 2 is hidden from theeyes. With such configurations, psychological burden of being imaged inthe bathroom can be reduced. The basic system configuration and thesignal processing are substantially the same as those of the firstembodiment.

An actual watch algorithm according to the third embodiment will bedescribed below with reference to FIGS. 7B and 7C. In the thirdembodiment, the biological information detection device illustrated inFIG. 7A (also referred to a “watch system”) is installed at the cornerof the bathroom and is configured to be able to monitor the entirebathroom as illustrated in part (a) of FIG. 7B. Human body detection(part (b) of FIG. 7B), body motion detection (part (c) of FIG. 7B), anddetection of abnormal heartbeat (part (d) of FIG. 7B) are performed byusing data of near-infrared images obtained by image capturing. If thereis no body motion after the human body is detected, for example, a firstalert (alert 1) is issued to a person who is taking a bath to draw theperson's attention. Further, if abnormal heartbeat is detected, a secondalert (alert 2) is issued to people located outside the bathroom, forexample. The operation of the watch system according to the thirdembodiment will be described in more detail below with reference to aflowchart of FIG. 7C.

FIG. 7C is a flowchart illustrating an operation of the watch systemaccording to the third embodiment. First, the first arithmetic circuit22 performs human body detection by using the same method as that usedin the first embodiment on the basis of data of captured near-infraredimages (S201). If a human body is detected, the process proceeds to stepS202 of body motion detection. At that time, data of the image used inhuman body detection is not stored in a storage device (e.g., the memory25 illustrated in FIG. 3B) and is overwritten by data of the next frameimage except for the data of the human body region. Since image datathat allows identification of an individual is not left, privacy isprotected.

Then, the second arithmetic circuit 23 detects a body motion bycomparing pieces of data of the detected human body region of aplurality of consecutive frames (step S202). For example, if no bodymotion is detected for a predetermined period (e.g., 30 seconds) ormore, the alert 1 is issued to the subject (step S203). The alert 1 canbe an alert, for example, “Are you awake? It is dangerous to sleep inthe bath. Please press the OK button if you are awake.” The alert 1 isintended to draw the attention of the subject and to check thecondition. When no body motion is detected, the second arithmeticcircuit 23 measures pulses (step S204). If the pulses are slow or nopulse is detected, the alert 2 is issued (step S205). The alert 2 is analert for people (such as a family member, a caregiver, or an ambulancecrew) located outside the bathroom. The alert 2 can be an alert intendedto request a target person set in advance in the system to check thesituation or to request a rescue via voice alert, telephone, or theInternet.

In accordance with the third embodiment, three-step detection of (1)human body detection, (2) body motion detection, and (3) heartbeatmeasurement can be performed with a simple system configuration.Accordingly, highly reliable watch can be implemented.

In the above-described example, the three-step detection of (1) humanbody detection, (2) body movement detection, and (3) heartbeatmeasurement is sequentially performed step by step; however, thedetection need not be performed in this order. For example, body motiondetection and heartbeat measurement may be performed in parallel afterhuman body detection. In this way, heartbeat of a person who is taking abath can be constantly monitored and an appropriate advance can be givento the person who is taking a bath. Death from drowning often occursbecause of dizziness (in the bathroom) that results from occurrence oforthostatic hypotension, which occurs in response to a change inheartbeat in response to contraction of blood vessels due to atemperature difference between a changing room and a bathroom and whichoccurs in response to a decrease in blood flow in the brain and theheart in response to an increase in blood flow at the body surface. Achange in the physical condition of a person who is taking a bath can bedetected in real time by constantly monitoring heartbeat as in the thirdembodiment. Upon detection of a change in the physical condition, afeedback is given to the person who is taking a bath. In this way, theabove-described accident can be avoided. For example, when an increasein heat rate is large, a message such as “You need to be careful fordizziness. When you stand up, please do so slowly while holding thehandrail.” can be issued.

Protection of privacy is especially important in a watch systeminstalled in private spaces, such as a bathroom, a lavatory, and abedroom. In the third embodiment, near-infrared images obtained with acamera are used only for human body detection, and the image data is notstored in a storage medium and is constantly overwritten by data of thenext frame after the human body detection processing. In addition, thesystem according to the third embodiment is designed so as not toinclude an image data output mechanism. Accordingly, it is impossible toobtain image data from outside. That is, the system is configured sothat privacy is not violated even the system is attacked by a malicioushacker. In addition, since the camera can be hidden so as not to be seenfrom the outside by the use of near-infrared light, watch can be donewithout causing the subject to have a feeling of being imaged. In thewatch system installed in private spaces, it is particularly essentialto ensure privacy in both the hardware aspect and the psychologicalaspect. The third embodiment implements watch at home while consideringthe privacy.

Fourth Embodiment

A system for measuring blood oxygen saturation in a non-contact mannerwill be described as a fourth embodiment. The main role of blood is toreceive oxygen at lungs, carries oxygen to tissues, receives carbondioxide from tissues, and carries carbon dioxide back to lungs.Approximately 15 g of hemoglobin is present in 100 ml of blood.Hemoglobin loaded with oxygen is called oxyhemoglobin (HbO₂), whereashemoglobin not loaded with oxygen is called hemoglobin ordeoxyhemoglobin (Hb). As illustrated in FIG. 22 , oxyhemoglobin anddeoxyhemoglobin have different light absorption properties.Oxyhemoglobin absorbs infrared light having wavelengths exceedingapproximately 830 nm relatively well, whereas deoxyhemoglobin absorbsred light (e.g., a wavelength of 660 nm) relatively well. There is nodifference between absorbance of oxyhemoglobin and absorbance ofdeoxyhemoglobin for near-infrared light having a wavelength of 830 nm.Accordingly, in the fourth embodiment, transmitting light having thesetwo wavelengths, that is, 660 nm and 830 nm, are measured. A ratio(oxygen saturation) between two types of hemoglobin can be determinedfrom a ratio between transmitting light of infrared light andtransmitting light of red light. Oxygen saturation is a value indicatinghow much hemoglobin in blood is loaded with oxygen. The oxygensaturation is defined by Equation below.

Oxygen saturation=C(HbO₂)/[(C(HbO₂)+C(Hb)]×100(%), where C(Hb) denotes adeoxyhemoglobin concentration, and C(HbO₂) denotes an oxyhemoglobinconcentration.

A living body includes a non-blood components that absorb light having awavelength of red to near-infrared light; however, a temporalfluctuation in light absorbance results mainly from hemoglobin inarterial blood. Accordingly, blood oxygen saturation can be measured ata high accuracy, based on a fluctuation in absorbance. Arterial bloodpumped out from the heart moves through a blood vessel as a pulse wave,whereas venous blood does not have a pulse wave. Light illuminating aliving body transmits through the living body after being absorbed ateach layer of the living body, such as arteries, veins, and non-bloodtissues. At that time, thickness of such tissues other than arteriesdoes not change over time. Accordingly, scattered light from the livingbody shows a temporal change in intensity in response to a change inthickness of an arterial blood layer due to the pulse. This changereflects a change in thickness of the arterial blood layer, and does notcontain the influence of venous blood and tissues. Thus, by focusingonly on the change in the scattered light component, informationconcerning arterial blood can be obtained. Also, heart rate can bedetermined by measuring a period of a change in the component over time.

FIG. 8 is a diagram illustrating a configuration of the system accordingto the fourth embodiment. The system includes light sources 101 and 102,image capturing devices 201 and 202, and a computer 20. The lightsources 101 and 102 are two array point light sources that are disposedat positions away from a living body 3 and emit a light beam of awavelength of near-infrared light (e.g. wavelength of 830 nm) and alight beam of a wavelength of red light (e.g. wavelength of 660 nm),respectively. The image capturing devices 201 and 202 are two camerascapable of recording an image of a living-body surface 4 illuminatedwith light. The computer 20 separates and measures an intensity ofdirectly reflected light reflected from the living-body surface 4 and anintensity of scattered light caused inside the living body from theobtained image and calculates biological information from the intensityof the directly reflected light and the intensity of the scatteredlight. In the fourth embodiment, the system includes the light sources101 and 102 with different wavelengths and the image capturing devices201 and 202 respectively corresponding to the light sources 101 and 102in order to measure blood oxygen saturation.

FIG. 9 is a diagram illustrating a configuration of the image capturingdevices 201 and 202. Each of the image capturing devices 201 and 202includes an optical system 5, which includes lenses, and a camera casing6. The camera casing 6 of the image capturing device 202 includes animage sensor 7 and a bandpass filter 802 that selectively passesnear-infrared light (wavelength of 830 nm). The camera casing 6 of theimage capturing device 201 includes the image sensor 7 and a bandpassfilter 801 that selectively passes red light (wavelength of 660 nm).

For example, a random dot pattern laser projector RPP017ES availablefrom Osela Inc. in Canada can be used as the light source 101. Thislaser light source is an 830-nm near-infrared laser light source andprojects a laser dot pattern including 57446 spots in a 45°×45° viewingangle. For example, a random dot pattern laser projector RPP016ESavailable from Osela Inc. in Canada can be used as the light source 102.This laser light source is a 660-nm red laser light source and projectsa laser dot pattern including 23880 spots in a 35°×35° viewing angle.

The computer 20 controls the image capturing devices 201 and 202 and thelight sources 101 and 102 so that the two image capturing devices 201and 202 operate together to simultaneously capture respective images. Inthis way, images based on light having two different wavelengths aregenerated by the image capturing devices 201 and 202 as illustrated onthe right in FIG. 9 , for example.

FIG. 10 is a diagram illustrating transmittance characteristics of thebandpass filters 801 and 802. The bandpass filter 801 has atransmittance characteristic that the transmission center wavelength is830 nm and the bandwidth is 10 nm. The bandpass filter 802 has atransmittance characteristic that the transmission center wavelength is660 nm and the bandwidth is 10 nm. The transmission center wavelengthsof the bandpass filters 801 and 802 respectively match central values ofwavelengths of the light sources 101 and 102. Accordingly, the imagecapturing device 201 obtains an image based on light having a wavelengthof 830 nm, whereas the image capturing device 202 obtains an image basedon light having a wavelength of 660 nm.

The first arithmetic circuit 22 of the computer 20 first extracts ahuman body region from a moving image as in the first embodiment. Thefirst arithmetic circuit 22 then performs data selection on pieces ofdata of pixels in that region by using a threshold to remove directlyreflected light components. Then, the first arithmetic circuit 22calculates an average of pixel values of the pixels in themeasurement-target region. The first arithmetic circuit 22 performs theabove process for each of the image capturing device 201 for 830 nm andthe image capturing device 202 for 660 nm. The averages thus obtainedindicate intensities of scattered reflected light from the living body3.

FIG. 11 is a diagram illustrating an example of temporal changes in theobtained scattered reflected light intensities. The reflected lightintensity fluctuates over time for both near-infrared light (wavelengthof 830 nm) and red light (wavelength of 660 nm). Here, let li(830) andli(660) respectively denote an intensity of light emitted from the lightsource 101 and an intensity of light emitted from the light source 102at the living-body surface 4, and let Δl(830) and Δl(660) each denote anaverage of the fluctuation component of the scattered reflected lightfrom the living body 3 over time. Blood oxygen saturation SpO₂ iscalculated using Equation below.

SpO₂ =a+b*(log(Δl(660)/li(660))/(log(Δl(830)/li(830))), where

a and b can be decided upon based on a relationship with measured valuesobtained by an existing pulse oximeter.

To check the accuracy of the measuring instrument, oxygen saturation ata fingertip instead of the forehead is measured using the systemaccording to the fourth embodiment. Specifically, oxygen saturation at afingertip is measured while blood flow is stopped by pressuring theforearm at a certain pressure (200 mmHg) using a belt (cuff) used tomeasure the blood pressure.

For comparison, a commercially available pulse oximeter to which afinger is inserted is attached to the index finger. Oxygen saturation atthe middle finger is measured in a non-contact manner by using thesystem according to the fourth embodiment. The above values a and b aredecided upon by the first measurement, and blood oxygen saturation SpO₂is measured by the second and subsequent measurements.

FIG. 12 illustrates a comparison result of the measured values obtainedusing the contact-type pulse oximeter and measured values obtained inaccordance with the fourth embodiment. Since both results substantiallymatch, FIG. 12 indicates that measurement is performed accurately. Inthe method of the fourth embodiment, not only blood oxygen saturationbut also heart rate can be simultaneously measured based on the pulsewaves illustrated in FIG. 11 .

It is known that stress and tiredness can be measured from thefluctuation or a frequency characteristic of the pulse wave. The use ofthe system according to the fourth embodiment makes it possible toestimate a mental condition, such as stress, and a physical condition ofa subject from the pulse wave in a non-contact manner.

Fifth Embodiment

A method for measuring blood oxygen saturation by using a single camerawill be described as a fifth embodiment. In fourth embodiment, twocameras are used and signals for the light sources with differentwavelengths are obtained by the respective cameras. This method has anadvantage in that existing cameras can be used. However, since imagesare captured by controlling two cameras to operate together, theconfiguration of the system becomes complex. Also, since the obtaineddata is individual pieces of video data for the two cameras,synchronized data processing becomes complex. To avoid suchcomplexities, in the fifth embodiment, a camera capable ofsimultaneously obtaining data of images for two wavelengths isimplemented.

FIG. 13 is a diagram illustrating a configuration of a biologicalinformation detection device according to the fifth embodiment. Thisbiological information detection device has a structure of a twin-lensstereo camera including two image capturing devices 201 and 202.Accordingly, herein, such a configuration is referred to as a “stereocamera configuration”. The biological information detection device,which is a camera, includes a light source 101 (wavelength of 830 nm),which is a first laser point light source, and a light source 102(wavelength of 760 nm), which is a second laser point light source. Thelight emitted by the light sources 101 and 102 and reflected by a livingbody 3 respectively passes through bandpass filters 801 and 802. Then,the propagating direction of the light is bent by mirrors 901 and 902 by90 degrees, and images are formed on imaging surfaces of image sensors701 and 702 via optical systems 501 and 502 (which include lenses),respectively. The bandpass filters 801 and 802 are narrow-band bandpassfilters that pass only light having a wavelength of 830±15 nm and lighthaving a wavelength of 760±15 nm that correspond to wavelengths of thetwo light sources 101 and 102, respectively.

In response to pressing of a shutter button 11, the two light sources101 and 102 switch on, and the image sensors 701 and 702 simultaneouslyobtain images of the living body 3. The obtained images are convertedinto images of a stereo-image format by an image processing processor(corresponding to the first arithmetic circuit 22 or the secondarithmetic circuit 23 in FIG. 3B), are subjected to image signalprocessing, and are accumulated in a storage device (corresponding tothe memory 25 in FIG. 3B). The following processing is substantially thesame as that of the third or fourth embodiment.

According to the fifth embodiment, by configuring an image capturingsystem as a single stereo camera, the entire system becomes compact, andthe configuration of the following signal processing system from imagesignal processing to calculation of oxygen saturation can be simplified.In this way, a simple and fast operation can be implemented.

For example, 760 nm and 830 nm, which are in a near-infrared range, canbe used as wavelengths of the two light sources. Since a difference inabsorbance between oxyhemoglobin and deoxyhemoglobin is larger for 660nm used in the fourth embodiment than that for 760 nm, oxygen saturationcan be measured more accurately for the wavelength of 660 nm. However,since the wavelength of 660 nm is in a visible light range, the use ofthis wavelength may impose a load on the subject. Further, since lightof a fluorescent lamp and a light-emitting diode (LED) illuminationcontains a component of the wavelength of 660 nm, measurement is easilyaffected by ambient light. In the fifth embodiment, the wavelength of760 nm is selected in consideration of such issues. Since an absorbancepeak of deoxyhemoglobin is at 760 nm, it is effective to use awavelength of 760 nm to 780 nm if the wavelength of the light sourcehaving a shorter wavelength is set in the near-infrared range. Thewavelengths used are not limited to the above ones, and may beappropriately selected in accordance with the usage and the useenvironment.

Sixth Embodiment

Another method for measuring blood oxygen saturation by using a singlecamera will be described as a sixth embodiment. In the fifth embodiment,the stereo camera configuration in which a single camera includes twooptical systems and two image sensors is employed. In the sixthembodiment, a system is employed which obtains two different imagescorresponding to two wavelengths with a single image sensor by dividingan image using a plurality of lenses. The configuration according to thesixth embodiment is referred to as a “stereo lens configuration”. Thesystem of the stereo lens configuration will be described with referenceto FIG. 14 .

FIG. 14 is a cross-sectional view schematically illustrating a part of abiological information detection device according to the sixthembodiment. Although not illustrated in FIG. 14 , the biologicalinformation detection device includes, for example, in a camera casing6, two light sources that project dot patterns formed by light havingtwo wavelengths of 830 nm and 760 nm. As illustrated in FIG. 14 , anoptical system 5 includes two optical systems 501 and 502 which are twosets of lenses. The optical systems 501 and 502 are designed to formrespective images at two different regions on the imaging surface of theimage sensor 7. Two narrow-band bandpass filters 801 and 802 that passlight having 830 nm and 760 nm corresponding to the wavelengths of thetwo light sources are disposed in front of the optical systems 501 and502, respectively.

With such a configuration, two images based on light having twowavelengths for the same time point can be obtained by using a singleimage sensor 7. The second arithmetic circuit 23 calculates blood oxygensaturation from the two images by using the method similar to that ofthe third to fifth embodiments. According to the sixth embodiment, sincea single image signal include information concerning two imagescorresponding to two different wavelengths for the same time point, thearithmetic processing becomes easier.

A result obtained by performing stress sensing by using the system ofthis stereo lens configuration will be described below. JapaneseUnexamined Patent Application Publication Nos. 6-54836 and 2008-237244disclose methods for detecting, with thermography, a decrease intemperature at a nose portion due to stress (nervousness) orconcentration has been proposed. The blood flow decreases at the noseportion due to a psychological change, and in response to the decreasein the blood flow, temperature decreases at the nose portion. Methodsfor detecting such a temperature change with thermography are commonlycarried out. A change in temperature at the face is caused by a changein blood flow. If a change in blood flow can be measured at a highaccuracy, stress sensing can be implemented that is more accurate andmore responsive than in the case of measuring a change in the surfacetemperature, which occurs as a result of a change in blood flow.

FIG. 15A is a diagram illustrating a result of stress sensing performedusing the biological information detection device according to the sixthembodiment. A cold-water load for immersing the right hand into coldwater (ice water) is imposed as stress. For comparison, a temperaturechange is measured using thermography at a nose portion and a cheekportion enclosed by dotted lines in FIG. 15B. FIG. 15A illustratesresults of this measurement. The temperature at the nose portiongradually decreases in about three minutes after the cold-water load isstarted to be imposed and becomes stable after decreasing byapproximately 1.2° C. FIG. 15A indicates that temperature returns to theoriginal level in about three minutes after the load is removed. Incontrast, temperature at the cheek portion is hardly affected by thecold-water load and is stable.

FIG. 15C is a diagram illustrating a change in blood flow and a changein blood oxygen saturation that are obtained by using the biologicalinformation detection device according to the sixth embodiment thatemploys the stereo lens configuration. Data for regions corresponding tothe nose portion and the cheek portion, which are denoted bydotted-lines in FIG. 15B, are extracted from data of the blood flow andthe oxygen saturation (SpO₂) at the face. A solid line denotes a changein the blood flow over time, whereas a dotted line denotes a change inthe oxygen saturation (ΔSpO₂) over time. As illustrated in FIG. 15C, theblood flow tends to decrease at the nose portion immediately after acold stimulus is started, which indicates the responsivity with respectto time is high. In contrast, the blood flow hardly changes at the checkportion. A decrease in the oxygen saturation is observed at the noseportion in response to a decrease in the blood flow, whereas the oxygensaturation hardly changes at the cheek portion.

As is apparent from the results, many pieces of data can be obtained bymeasuring blood flow and oxygen saturation at different portions of theface. An emotion, a physical condition, and a concentration degree canbe detected at a high accuracy based on these pieces of data. The changein blood flow due to influence of the autonomic nervous system differsfrom portion to portion of the face. Thus, it is especially important tomeasure a change in blood flow at a specific portion by using a camera.At that time, the accuracy of the measurement can be increased byperforming, at the same time, measurement at a portion where blood flowhardly changes and using the result as a reference.

Seventh Embodiment

Another method for measuring blood oxygen saturation by using a singlecamera will be described as a seventh embodiment.

FIG. 16 is a sectional view schematically illustrating a configurationof a biological information detection device according to the seventhembodiment. The biological information detection device includes astereo adapter 10 attachable to an optical system 5, which includesordinary lenses. The stereo adapter 10 is an attachment including fourmirrors 151, 152, 153, and 154 and two bandpass filters 801 and 802. Theuse of the stereo adapter 10 allows two images corresponding to twowavelengths to be formed at two different regions on the imaging surfaceof an image sensor 7. This configuration is referred to as a “stereoadapter configuration”.

In the stereo adapter configuration, two different images correspondingto two wavelengths can be obtained by a single image sensor 7 by usingtwo sets of facing mirrors. Although not illustrated in FIG. 16 , twolight sources that respectively emit light having two wavelengths of 830nm and 760 nm are included in a camera casing 6. The stereo adapter 10is attached to an end of the optical system 5. The two sets of mirrors(a set of mirrors 151 and 152 and a set of mirrors 153 and 154) bend thelight path twice to guide the light to the optical system 5. Thenarrow-band bandpass filters 801 and 802 that respectively pass lighthaving the wavelengths of 830 nm and 760 nm corresponding to thewavelengths of the light sources are disposed between the optical system5 and the mirrors 151, 152, 153, and 154.

This biological information detection device is able to obtain images oftwo wavelengths for the same time point by using a single image sensor7. The basic concept is the same as that of the sixth embedment. Sincethe stereo lens configuration can make the optical system small, theentire system can be advantageously made small. In contrast, with thestereo adapter configuration, the entire system becomes larger but apowerful camera lens can be used and the resolution can be improved.Also, lenses of different magnifications and zoom lenses can be used.The stereo adapter configuration advantageously increases the degree offreedom of the system.

A study for detecting an emotion of a person by using the biologicalinformation detection device according to the seventh embodiment hasbeen carried out. As described in the sixth embodiment, a feeling oremotion such as stress of a person can be stably detected based on bloodflow. In response to a change in a feeling or emotion of a person, theautonomic nervous system becomes more active and blood flow on the skinsurface changes. As a result of this change in blood flow, facial colorchanges. People detect an emotion or a physical condition of a targetperson from a subtle change in facial color without any difficulty. Itis considered that a reason why a great doctor can diagnose a patient'sphysical condition and a cause of a disease by just looking at thepatient's face is that such a doctor can identify a physical change fromthe subtle change in color of the patient's face. In addition, it issaid that, when a person who is good at reading the situation reads afeeling of a counterpart, a subtle change in facial color as well as asubtle change in facial expression play an important role. Further, tomake a situation natural and real in fields showing remarkableprogresses recently, such as game, animation, and computer graphics,studies for subtly changing the facial color of a human character arewidely carried out. As is apparent from these examples, the facial colorrepresents an emotion and a physical condition of a person, and anattempt to read a feeling by measuring the facial color has been studied(see, for example, Kuroda and one other, “Analysis of facial color andskin temperature in emotional change and its synthesis of facial color”,Human interface 1(1), pp. 15-20, Feb. 16, 1999). However, such anattempt to directly measure an emotion from facial color is not suitablefor practical use because stable measurement is difficult. This isbecause a change in facial color differs from person to person and achange in facial color is subtle and is strongly influenced bydisturbance light and a camera. Thus, stable measurement is difficult. Amethod for stably and highly accurately detecting an emotion by ameasure other than measuring a change in facial color is desired.

It is known that facial color of a person is mainly decided by an amountof melanin contained in the skin surface (dermis) and concentrations ofhemoglobin (oxyhemoglobin and deoxyhemoglobin) in blood. Since theamount of melanin does not fluctuate in a short time (changes due to afactor such as aging or tanning), a change in emotion can be stablymeasured by measuring blood flow. In the seventh embodiment, instead ofmeasuring facial color, blood flow of oxyhemoglobin and deoxyhemoglobinthat change the facial color is directly measured to detect a change inemotion. As described in the sixth embodiment, a change in blood flowdiffers from portion to portion of the face because an impact of theinfluence of the autonomic nervous system differs from portion toportion of the face. For example, the nose portion is easily influencedby the autonomic nervous system because lots of arteriovenousanastomosis is located at the nose portion, whereas the forehead portionis hardly influenced by a skin blood vessel contraction nerve. Thesecond arithmetic circuit 23 according to the seventh embodimentdetermines blood flows at a plurality of different portions bycomputation and compares the obtained blood flows with each other,thereby being able to detect a change in emotion at a high accuracy.

Measurement of a change in blood flow in response to an emotional changewill be described below. The camera of the stereo adapter configurationillustrated in FIG. 16 is used to measure blood flow. A subject sits infront of the camera, and an image of the subject's face is captured withthe camera. Color images of the face of the subject are obtained whileshowing the subject a video image that induces, from the secured state,feelings such as fear, laughter, surprise, and disgust. An emotionalchange is read based on a change in scene in the video image and thefacial change in the color images, and a change in blood flow at thetime of such a change is measured. Blood flow is measured at the noseportion and the forehead portion as indicated by dotted lines in FIG.17A.

FIG. 17B is a diagram illustrating a change in total blood flow(oxyhemoglobin and deoxyhemoglobin) over time and a change in thepercentage of oxyhemoglobin blood flow (oxygen saturation) over timewhen an emotion causing laughter is induced. FIG. 17B indicates that thevalue of the total blood flow and the value of the blood oxygensaturation greatly change in response to the emotional change causinglaughter. Similar examinations are performed for other emotions. FIG. 18illustrates the result. FIG. 18 is a diagram obtained by plotting arelationship between blood flow and oxygen saturation that occurs inresponse to each emotional change by setting oxygen saturation as thehorizontal axis and setting the total blood flow as the vertical axis. Achange in the total blood flow and a change in the blood oxygensaturation are determined by calculation for the case where emotionsother than laughter, such as sadness, surprise, depression, fear,disgust, anger, concentration, and happiness, are induced. The samemeasurement is performed for twelve subjects. FIG. 18 illustrates theaverage of the experiment results obtained for the twelve subjects.Although there is a variation among individuals, the change in the totalblood flow and the change in the blood oxygen saturation have showed thesimilar tendencies for almost all the subjects. This result indicatesthat an emotional change can be detected from at least one of blood flowand oxygen saturation.

As illustrated in FIG. 17B, a relationship between oxygen saturation andblood flow differs from portion to portion of the face. Accordingly,highly accurate emotion sensing can be performed by determining bloodflow and oxygen saturation at a plurality of portions of the face. Inthe emotion sensing test performed in the seventh embodiment,measurement is performed at three portions, i.e., at the forehead, thecheek, and the nose. A change in oxygen saturation and a change in bloodflow in response to an emotional change differ among the forehead, thecheek, and the nose. Accordingly, an emotional change can be detectedhighly accurately by creating in advance a table indicating arelationship between a change in oxygen saturation and a change in bloodflow at each portion and calculating a correlation with the actuallymeasured values of oxygen saturation and blood flow.

Eighth Embodiment

A method for measuring blood oxygen saturation by using a single camerawithout dividing an image by an optical system will be described as aneighth embodiment. In the third to seventh embodiments, the methods fordividing light from two light sources corresponding to two wavelengths,performing sensing, and determining biological information, such asoxygen saturation, by computation have been described. A biologicalinformation detection device according to the eighth embodiment obtains,with an image sensor, two image signals for different wavelengthswithout dividing an image.

FIG. 19A is a diagram schematically illustrating a configuration of thebiological information detection device according to the eighthembodiment. This biological information detection device separates twoimages corresponding to two wavelengths by using an image sensor 703instead of by using the optical system. Although illustration of pointlight sources is omitted in FIG. 19A, two light sources thatrespectively emit light having a wavelength of 830 nm and light having awavelength of 760 nm are included in a camera casing 6. A bandpassfilter 8 that passes light having a wavelength longer than or equal to730 nm and shorter than or equal to 850 nm is disposed in front of anoptical system 5, which includes lenses, of the camera. The bandpassfilter 8 cuts visible light and infrared light having long wavelengths.Light that has passed through the bandpass filter 8 forms an image onthe imaging surface of the image sensor 703 via the optical system 5.Unlike ordinary image sensors, the image sensor 703 used in the eighthembodiment includes two kinds of bandpass filters that passnear-infrared light.

FIG. 19B is a diagram illustrating a plurality of filters that face aplurality of photodetector cells disposed on the imaging surface of theimage sensor 703. The image sensor 703 includes filters IR1 thatselectively pass light having a wavelength of 680 nm to 800 nm andfilters IR2 that selectively pass light having a wavelength of 800 nm orlonger. The filters IR1 and IR2 are arranged in a checkered pattern. Thelower image in FIG. 19B is a diagram illustrating an example ofwavelength dependency of transmittance of the filters IR1 and IR2. Theimage sensor 703 detects, with a plurality of photodetector cells (alsoreferred to as pixels), two images based on light having 760 nm and 830nm which are wavelengths of the two light sources.

The first arithmetic circuit 22 (FIG. 3B) separately reads data obtainedby the plurality of photodetector cells of the image sensor 703 for thewavelength of 760 nm and data obtained by the plurality of photodetectorcells for the wavelength of 830 nm. The first arithmetic circuit 22detects a human body region in each of the images. As illustrated inFIG. 19C, the second arithmetic circuit 23 (FIG. 3B) then generates animage for the wavelength of 760 nm and an image for the wavelength of830 nm by adding, to the corresponding data, data for lacking pixels byinterpolation. The second arithmetic circuit 23 calculates blood flowand oxygen saturation from these two images. Since these two imagescompletely coincide with each other, calculating blood flow and oxygensaturation from these images is easier than calculation using twodifferent images. However, since the filtering performance of thefilters is lower than that achieved in the case where bandpass filterscorresponding to respective light sources are used, there is a concernabout occurrence of color mixing between the light sources in thismethod.

Ninth Embodiment

A biological information detection device capable of obtaining not onlytwo images corresponding light sources with two wavelength but also acolor image without dividing an image will be described as a ninthembodiment.

FIG. 20A is a diagram illustrating a configuration of the biologicalinformation detection device according to the ninth embodiment. Althoughillustration of point light sources is omitted also in FIG. 20A, twolight sources that respectively emit light having a wavelength of 830 nmand light having a wavelength of 760 nm are included in a camera casing6. In the ninth embodiment, to obtain a color image, no bandpass filteris disposed in front of an optical system 5, which includes lenses.Visible light and light emitted by the leaser light sources form imageson an imaging surface of an image sensor 704 via the optical system 5.Unlike ordinary image sensors, the image sensor 704 used in the ninthembodiment includes photodetector cells that obtain a color image andtwo kinds of photodetector cells that obtain near-infrared images.

FIG. 20B is a diagram illustrating a plurality of bandpass filters (orcolor filters) disposed on the imaging surface of the image sensor 704.The lower image in FIG. 20B illustrates wavelength dependencies ofrelative sensitivities of pixels that face corresponding filters. Asillustrated in FIG. 20B, three types of color filters (R, G, and Bfilters) that respectively pass red light, green light, and blue light,filters IR-1 that pass light having 650 nm or longer, and filters IR-2that pass light having 800 nm or longer are arranged on the imagingsurface. An array in which two G filters are disposed diagonallyadjacent to each other and R and B filters are disposed on the oppositediagonal side is the same as the Bayer array of ordinary image sensors.This image sensor 704 differs from ordinary image sensors in that twofilters IR-1 and IR-2 are arranged next to a basic unit of four filtersarranged in the Bayer array.

The filter IR1 of the eighth embodiment and the filter IR-1 of the ninthembodiment have different transmission wavelength ranges. The filter IR1of the eighth embodiment is a relatively narrow-band filter that passeslight having a wavelength range from 650 nm to 800 nm. In contrast, inthe ninth embodiment, a filter that passes light having a wavelengthrange of 650 nm or longer is used to simplify the manufacturing processof the image sensor 704. However, the configuration is not limited tothis one, and the filter described in the eighth embodiment can also beused. The filter IR-1 of the ninth embodiment is sensitive to both 760nm and 830 nm. Accordingly, the second arithmetic circuit 23 calculatesa signal corresponding to 760 nm by subtracting the signal of thephotodetector cells that face the filter IR-2 from the signal of thephotodetector cells that face the filter IR-1 and then determines bloodoxygen saturation. Consequently, an image (color image) of red, blue,and green, an image for the wavelength of 760 nm, and an image for thewavelength of 830 nm are determined by the image sensor 704 asillustrated in FIG. 20C.

In this configuration, color mixing is more likely to occur than in theeighth embodiment. However, a color image and information regardingblood flow and blood oxygen saturation can be simultaneously obtained bythe simple system using a single camera. An advantage of this system isthat visible-light-based images without parallax and near-infraredimages can be obtained since a single camera is used to capturevisible-light-based images and near-infrared images. This isparticularly effective in usages in which a visible-light-based imageand near-infrared images are superimposed for display.

In the ninth embodiment, an example of a configuration of a biologicalinformation sensing camera that uses a multi-spectral sensor thatsupport five wavelengths including two wavelengths in an infrared rangeand three wavelengths (red, blue, and green) in a visible light rangehas been described. With the human body detection camera described inthe first embodiment, capturing of a color image and human bodydetection can be performed through measurement on four wavelengthsincluding one wavelength in an infrared range and three wavelengths(red, blue, and green) in a visible light range. The multi-spectralsensor (illustrated in FIG. 20D, for example) having four types of colorfilters corresponding to four wavelengths is usable for such a purpose.A color filter is disposed such that a near-infrared (IR) pixel isassigned to one pixel out of two green pixels of the Bayer array that iscommonly used in the image sensor. In the ninth embodiment, a camera fora system that switches on a near-infrared illumination of 850 nm isassumed, and a filter that selectively passes the wavelength of 850 nmis selected as a near-infrared filter. The use of such a camera makes itpossible to use a single camera system as an ordinary color camera and aliving body detection camera. Consequently, only one surveillance camerais needed, and it is easier to extract a color image of a portion inwhich a person is detected than in the case where two cameras are used.In the ninth embodiment, a color filter for 850 nm is used; however, thenear-infrared filter may be changed in accordance with the near-infraredlight source used.

OTHER EMBODIMENTS

While the embodiments of the present disclosure have been describedabove by way of example, the present disclosure is not limited to theabove embodiments and can be variously modified. The process describedin each of the above embodiments may be applied to other embodiments.Examples of the other embodiments will be described below.

In the embodiments above, laser light sources are used as the arraypoint light sources; however, light sources of other types may be used.For example, less costly LED light sources may be used. However, lightemitted by the LED light source has a lower straightness than thatemitted by the laser light source and is more likely to spread.Accordingly, when LED light sources are used, a dedicated condensingoptical system may be used or a distance between an image-capturingtarget and a camera may be limited.

The biological information detection device may include an adjustmentmechanism that adjusts focus of the optical system. Such an adjustmentmechanism can be implemented by, for example, a motor (not illustrated)and the control circuit 26 illustrated in FIG. 3B. Such an adjustmentmechanism adjusts focus of the optical system to maximize contrast of adot pattern image projected onto a target by the light source. With thisconfiguration, accuracy of calculation of contrast described in thefirst embodiment improves.

The first arithmetic circuit 22 detects a living body region by using animage signal output from an image sensor. If a plurality of living bodyregions (a face region and a hand region of different persons or thesame person) are detected in the image at that time, the firstarithmetic circuit 22 may select a living body region that should bedetected, on the basis of the size or shape of the detected regions.

The second arithmetic circuit 23 may generate information concerning theepidermis including information concerning at least one of a melaninconcentration, presence or absence of a spot, and presence or absence ofa bruise on the basis of the image signal. As described above, theepidermis contains melanin that strongly absorbs light. A spot and abruise are caused as a result of an increase in melanin. Accordingly, amelanin concentration, a spot, and a bruise can be detected based on anintensity distribution of light from the living-body surface. Forexample, the second arithmetic circuit 23 may extract, from the imagesignal, directly reflected light components from the surface of a livingbody and may generate information concerning the epidermis includinginformation concerning at least one of a melanin concentration, presenceor absence of a spot, and presence or absence of a bruise on the basisof the directly reflected light components. The directly reflected lightcomponents can be obtained by determining whether contrast exceeds apredetermined threshold as in the first embodiment or by removing alow-frequency component from the image signal, for example.

In the present disclosure, the double camera configuration using twocameras (FIG. 8 ), the stereo camera configuration (FIG. 13 ) in whichone camera includes two optical systems and two image sensors, thestereo lens configuration (FIG. 14 ) using two sets of lenses and oneimage sensor, the stereo adapter configuration (FIG. 16 ) using a lensadapter, one lens, and one image sensor, the configuration (FIG. 19A andFIG. 20A) that divides an image using the image sensor have beendescribed. As already described, since each configuration has advantagesand drawbacks, an appropriate configuration can be selected inaccordance with the usage.

As described above, according to the embodiments of the presentdisclosure, not only heart rate and blood flow but also blood oxygensaturation can be measured without restraining a subject and withoutplacing a detection device, such as a sensor, in contact with thesubject. An emotion or a physical condition of the subject can also beestimated from measured values of blood flow and oxygen saturation atdifferent portions of the subject.

What is claimed is:
 1. A system comprising: a light source that, inoperation, projects dots onto a target, the dots being formed by firstlight; a first photodetector that, in operation, detects second lightresulting from the projection of the dots onto the target; and a circuitthat in operation, performs an individual authentication, wherein theindividual authentication includes at least the following step(i) andstep(ii): step(i) determining whether the target is a living body or notbased on the second light, and step(ii) performing a biometricauthentication of the target.
 2. The system according to claim 1, in thestep(i), wherein the circuit, in operation, determines whether thetarget is a living body or the duplicate of a living body.
 3. The systemaccording to claim 1, wherein the second light includes: reflected lightthat is reflected by the target; and scattered light that exits frominside of the target after entering the inside of the target and beingscattered, and in the step(i), the circuit, in operation, determineswhether the target is a living body or not, based on the directlyreflected light and the scattered light.
 4. The system according toclaim 3, in the step(i), wherein the circuit, in operation, determineswhether the target is a living body or not, based on a ratio between thedirectly reflected light and the scattered light.
 5. The systemaccording to claim 1, wherein the first light has a wavelength longerthan or equal to 650 nm and shorter than or equal to 950 nm.
 6. Thesystem according to claim 1, wherein the light source projects aplurality of discretely arranged dot pattern onto the target.
 7. Thesystem according to claim 1, in the step(ii) wherein the biometricauthentication includes at least one piece of authentication selectedfrom the group consisting of fingerprint authentication, irisauthentication, and vein authentication.
 8. The system according toclaim 1, wherein the circuit, in operation, determines that theindividual authentication is successful when it is determined that thetarget is a living body and the biometric authentication is successful.9. The system according to claim 1, wherein the target is a face of aperson, and in the step(i) the circuit, in operation, determines whetherthe face is a part of a living body or not.
 10. The system according toclaim 1, wherein the first photodetector is a camera that captures animage of the target, and both of the determination whether the target isa living body or not and the biometric authentication are performedbased on the image captured with the camera.
 11. A method comprising:causing a light source to project dots onto a target, the dots beingformed by first light; causing a first photodetector to detect secondlight resulting from the projection of the dots onto the target; andperforming an individual authentication that includes at least thefollowing step(i) and step(ii): step(i) determining whether the targetis a living body or not based on the second light; and step(ii)performing a biometric authentication of the target.